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ABSTRACT 


The  research  reported  in  this  thesis  concentrates  on  a subclass  of  visual  information  processing 
referred  to  as  vtrificatim  vision  (abbreviated  VV).  VV  uses  a model  of  a scene  to  locate  objects 
of  interest  in  a picture  of  the  Kene.  The  characteristia  that  distinguish  VV  from  the  other 
types  of  visual  information  processing  are:  (I)  the  system  has  a great  deal  of  prior  knowledge 
about  the  type,  placement,  and  appearance  of  the  objecu  that  form  the  scene  and  (2)  the  goal  is 
to  verify  and  refine  the  location  of  one  or  nrare  objects  in  the  Kene.  VV  includes  a significant 
portion  of  the  visual  feedback  tasks  required  within  programmable  assembly.  For  example, 
locating  a Krew  hole  and  determining  the  relative  displacement  between  a Krew  and  the  Krew 
hole  are  both  VV  tasks. 

There  are  several  types  of  Information  available  in  VV  tuks  that  can  facilitate  the  solution  of 
such  tasks:  a nradel  of  each  object  in  the  scene,  a set  of  initial  constraints  on  the  locations  of  the 
objects,  and  a set  of  previous  pictures  of  this  scene  or  similar  scenes.  Additional  information  can 
be  obtained  by  applying  visual  operators  to  a current  picture  of  the  Kene.  [continued  next  page] 
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How  can  all  of  this  Information  b«  used  to  minimize  the  amount  of  work  required  to  perform  a 
task?  Two  steps  are  involved  In  answering  this  question:  (I)  formalize  the  types  of  tasks, 
available  information,  and  quantities  of  interest  and  (2)  formulate  combination  rules  that  use  the 
available  information  to  estimate  the  quantities  of  interest.  The  combination  rules  that  estimate 
confidences  are  based  upon  Bayes'  theorem.  They  are  designed  to  combine  the  results  of 
operators  that  are  not  completely  reliable,  l.e.,  operators  that  may  And  any  one  of  several  known 
features  or  a surprUt.  The  combination  rules  that  estimate  precisions  are  based  upon  a least- 
squares  technique.  They  use  the  expected  precisions  of  the  operators  to  check  the  structural 
consistency  of  a set  of  matches  and  to  estimate  the  resulting  precisions  of  the  points  of  Interest. 

An  Interactive  VV  system  based  upon  these  Ideas  hu  been  implemented.  It  helps  the 
programmer  select  potentially  useful  operator/feature  pairs,  provides  a training  session  to  gather 
statistics  on  the  behavior  of  the  operators,  automatically  ranks  the  operator/feature  pairs 
according  to  their  expected  contributions,  and  performs  the  desired  task.  The  VV  system  has 
also  been  interfaced  to  the  AL  control  system  for  the  mechanical  arms  and  has  been  tested  on 
tasks  that  Involve  a combination  of  touch,  force,  and  visual  feedback. 

This  thesis  was  submitted  to  the  Department  of  Computer  Science  and  the  Committee  on 
Graduate  Studies  of  Stanford  Universi^  in  partial  fUMIIment  of  the  requiremenu  for  the  degree 
of  Doctor  of  Philosophy. 
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PREFACE 


The  work  reported  in  this  thesis  was  performed  while  the  author  was  a member  of  the 
hand/eye  group  at  the  Stanford  Artificial  Intelligence  Project.  The, group  has  traditionally 
followed  the  philosophy  that  one  of  the  most  important  stages  in  testing  a theory  is  to  build 
the  necessary  hardware  and  software  and  make  sure  that  the  combination  performs  as 
desired  In  line  with  that  philosophy  a verification  vision  (abbreviated  VV)  system  has  been 
implemented  that  incorporates  most  of  the  ideas  discussed  in  this  thesis.  It  has  been  used  to 
perform  several  inspection  and  location  tasks.  It  has  also  been  interfaced  to  the  mechanical 
arm  and  the  combination  has  been  used  to  perform  a few  tasks  that  involve  both 
manipulation  and  visual  feedback. 

The  current  implementation  suffers  from  a well-known  disease  of  programming- 
unplanned  growth.  Because  of  that,  a few  of  the  ideas,  such  as  the  heuristic  to  determine 
conservative  distributions,  have  been  implemented  as  separate  programs.  All  of  the  examples 
used  in  the  thesis  are  results  from  the  VV  system  or  one  of  these  auxiliary  programs. 
Appendices  IV  and  V arc  extensive  traces  of  the  interaction  between  a programmer  and  the 
VV  system  as  the  programmer  prepares  VV  programs  to  perform  different  tasks.  Comments 
have  been  added  to  these  traces  and  a few  minor  changes  have  been  made  to  clarify  the 
explanations. 


The  thesis  is  a combination  of  theory  and  practice.  Chapters  three,  four,  and  five 
present  the  theory;  the  rest  of  the  chapters  present  the  practical  motivation  and  system  aspects 
The  casual  reader  can  pick  up  the  main  ideas  by  reading  the  introduction  (chapter  I),  the 
motivation  chapter  (chapter  2),  the  two  traces  (appendices  IV  and  V),  and  the  conclusion 
(chapter  7).  A more  serious  reader  may  also  want  to  look  at  the  traces  and  conclusion  before 
starting  chapters  three,  four,  and  five. 


Iv 


ACKNOWLEDGEMENTS 


I feel  overwhelmed  by  the  number  of  people  who  have  directly  helped  me  as  I have 
worked  on  this  thesis.  I want  to  thank  them  all. 

First,  I would  like  to  thank  my  advisor,  Jerry  Feldman,  for  his  continual  enthusiasm, 
encouragement,  and  advice.  His  enthusiasm  helped  start  the  hand/eye  project,  without  which 
there  would  not  have  been  the  intellectual  environment  and  facilities  that  have  been  so 
important  to  this  work.  Other  members  of  the  hand/eye  group  whom  I have  known  and 
worked  with  include:  Lou  Paul,  Tom  Binford,  Vic  Scheinman,  Bruce  Baumgart,  Lynn  Q_uam, 
Marsha  Jo  Hannah,  Ram  Nevatia,  Randy  Davis,  Karl  Pingle,  Arthur  Thomas,  Bruce 
Anderson,  Eiichi  Miyamoto,  Charlie  Wihon,  Joe  Zingheim,  Russ  Taylor,  Ray  Finkel,  Bruce 
Shimano,  Don  Cennery,  Hans  Moravec,  Mike  Roderick,  Ron  Goldman,  Dave  Grossman,  and 
Shahid  Mujtaba.  There  are  also  several  past  members  of  the  group  whom  I’ve  never  met, 
but  whose  code  I’ve  used.  1 hank  you  all  for  making  it  possible  and  exciting  to  work  in  the 
field  of  robotics. 

I would  especially  like  to  thank  Lou  Paul  for  his  insights  and  understanding.  His 
attitude  and  the  attitude  of  the  other  people  at  the  A. I.  Lab  meant  a great  deal  to  me 
throughout  my  stay  at  Stanford. 

I would  like  to  thank  Tom  Binford  for  his  guidance  and  his  many  careful  readings  of 
this  manuscript  as  it  was  being  written. 

I would  like  to  thank  Dave  Barstow  for  his  friendship  that  has  included  moral  and 
emotional  support  in  addition  to  physical  and  intellectual  exercise.  Thank  you. 

I would  like  to  thank  my  family;  my  wife,  Jomary;  my  mother,  father,  brother,  and 
sister;  for  their  encouragment  and  support  at  the  many  decision  points  along  the  way. 

Finally,  I would  to  thank  the  Hertz  Foundation,  the  National  Science  Foundation,  and 
the  Advanced  Research  Projects  Agency  for  their  financial  support. 


V 


TABLE  OF  CONTENTS 


Chapter  1.  INTRODUCTION 1 

Chapter  2.  MOTIVATION  12 

Section  I.  CHECKING  FOR  A SCREW 12 

Section  2.  LOCATING  A HOLE 21 

Section  3.  SUMMARY  OF  RE(iUIRED  FACILITIES  24 

Chapters.  EXECUTION-TIME  COMBINATION  RULES  FOR  INSPECTION . . . . 30 

Section  I.  OPERATOR  VALUE  INFORMATION 30 

Section  2.  KNOWN  ALTERNATIVES  FOR  A FEATURE 44 

Section  3.  SURPRISES 51 

Section  4.  MULTIPLE-VALUED  OPERATORS  56 

Section  5.  POSITION  INFORMATION 58 

Section  6.  ASSUMPTIONS 70 

Chapter  4.  EXECUTION-TIME  COMBINATION  RULES  FOR  LOCATION  . . . . 81 

Section  I . DETERMINING  PRECISION  82 

Section  2.  CONFIDENCE  IN  THE  PRECISION  87 

Chapter  5.  PLANNING-TIME  COMBINATION  RULES 93 

Section  I.  RANKING  FEATURES  BY  VALUE 94 

Section  2.  KNOWN  ALTERNATIVES  AND  SURPRISES 96 

Section  3.  COST  INFORMATION 99 

Section  4.  LEAST-SQUARES  CULLING 100 

Section  5.  INSPECTION  104 

Section  6.  PRECISION 106 

Section  7.  CONFIDENCE  IN  THE  PRECISION  108 

Section  8.  EXPECTED  COST 108 

Chapters.  PROGRAMMING  TIME  AND  TRAINING  TIME 110 

Section  1.  STATE  THE  TASK 112 

Section  2.  POSITION  AND  CALIBRATE  THE  CAMERA. 116 

Section  3.  CHOOSE  POTENTIAL  OPERATOR/FEATURE  PAIRS 126 

Section  4.  GATHER  STATISTICS 147 

Chapter  7.  CONCLUSIONS  AND  EXTENSIONS  156 

Section  1.  FRAMEWORK  FOR  VV  156 

Section  2.  EASE  OF  PROGRAMMING 158 


vl 


BIliLlOGRAPI  lY  , . . . 161 

APPENDIX  1 ASSIGNMENTS  FOR  TWO  OPERATORS 172 

APPENDIX  11  CEST  KNOWN  ALTERNATIVES 176 

APPENDIX  III  EXPICTED  LOG-LIKELIHOOD  RATIO  FORMULA 179 

APPENDIX  IV.  SCREW-INSPECTION  EXAMPLE 186 

Section  1 1 R ACE  OF  A SCREW-INSPECTION  TASK 186 

Seciiui.  2 1 ASK  FILE  FOR  THE  SCREW-INSPECTION  TASK 200 

Section  ?.  CALlliRATION  WHU  RESPECT  TO  THE  SCREW  DISPENSER  . . 202 

Sections  INTERFACE  TO  THE  MECHANICAL  ARM 206 

APPENDIX  V.  HOLE  LOCATION  EXAMPLE 210 

APPENDIX  VI  CURVES  AS  FEATURES  22S 

Section  1.  P.ASIC  APPROACH 226 

Section  2.  LOCAL  TFSTS  FOR  A POINT  ON  A CURVE  229 

Section  ?.  SPECIFICATION  OF  A CURVE  230 

Section  S REPRESENTATION  OF  A CURVE  235 


Page  I 


CHAPTER  1 
INTRODUCTION 


Verification  vision  is  a type  of  visual  information  processing  that  uses  a model  of  a 
scene  to  locate  objects  of  interest  in  a picture  of  the  scene.  The  characteristics  that  distinguish 
verification  vision  (abbreviated  VV)  from  the  other  types  of  visual  information  processing 
are.  (1)  the  system  has  a great  deal  of  prior  knowledge  about  the  type,  placement,  and 
appearance  of  the  objects  that  form  the  scene  and  (2)  the  goal  is  to  verify  and  refine  the 

location  of  one  or  more  objects  in  the  scene.  The  following  situation  illustrates  a typical  use 

of  VV: 

During  the  assembly  of  a pump,  a mechanical  arm 

places  the  pump  base  In  a vise.  The  next  step  Is  to 

Insert  an  aligning  pin  Into  one  of  the  screw  holes  In  the 
base.  But  the  location  of  the  screw  hole  Is  not  known 
precisely  enough  to  Insert  the  pin  directly.  The 
programmable  assembly  program  needs  to  Improve  Its 

estimate  for  the  location  of  the  hole.  W Is  one  way  to 

accomplish  this  subtask. 

In  this  task  the  pump  base  may  be  mispositioned  to  the  extent  of  perhaps  plus  or  minus  half 
an  inch  and  rotated  plus  or  minus  fifteen  degrees,  but  there  will  not  be  any  big  surprises:  the 
base  will  not  be  upside  down  or  at  the  other  end  of  the  workstation. 

The  class  of  VV  tasks  can  be  placed  in  perspective  by  comparing  it  to  other  types  of 
visual  information  processing.  Baumgart  distinguishes  three  types:  description,  recognition, 
and  verification  (see  [Baumgart  74b]).  These  types  can  be  conveniently  defined  in  terms  of 
three  factors:  prior  knowledge  about  what  can  be  in  a scene  (and  can  therefore  appear  in  a 
picture  of  the  Kene),  prior  knowledge  about  where  things  might  be  with  respect  to  the  camera 
(and  hence  where  they  might  appear  in  a picture),  and  the  goal  of  the  task.  The  three  types 
of  tasks  are 
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DESCRIPTION 

Prior  knowledge  only  includes  the  types  of  features  that  comprise 
the  objects  and  how  to  build  complex  models  from  these  features; 
the  identity  and  position  of  the  objects  are  unknown;  the  goal  is 
to  build  a model  that  describes  the  scene. 

If  you  have  never  seen  a telephone,  but  you 
know  about  colors,  shapes,  and  sizes,  you  might  describe 
a standard,  black  telephone,  like  this:  it  is  black 

and  about  half  the  size  of  a shoe  box;  there  is  a 
dumbbell-shaped  crossbar  on  top,  near  the  bock;  it  has 
a round,  disk  with  several  holes  in  it  on  the  top,  near 
the  front;  the  ten  digits  are  arranged  in  an  arc  of  a 
circle  underneath  the  holes  in  the  disk. 

RECOGNITION 

Prior  knowledge  includes  a fixed  set  of  possible  object  models  and 
occasionally  a few  constraints  on  where  the  objects  might  occur; 
the  goal  is  to  identify  an  object  in  the  scene  and  possibly 
quantify  a few  paramiters  about  it. 

If  you  know  about  telephones,  what  they  look 
like,  where  they  normally  are,  and  what  they  are  used 
for,  you  might  identify  a telephone  in  an  office  like 
this:  you  will  probably  first  look  on  the  desk  for 

something  black  and  about  half  the  size  of  a shoe  box. 
There  may  bo  several  objects  on  the  desk  that  you 
recognize:  books,  pencils,  a coffee  cup,  and  a telephone. 
Having  recognized  the  phone  you  could  roughly  describe 
its  location:  it  is  on  the  far  left  corner  of  the  desk, 
almost  against  the  back  wall.  Garvey  has  written  a 
program  that  performs  this  type  of  recognition  for 
telephones  and  other  office  equipment  (see  [Garvey 
76]). 

VERIFICATION 

Prior  knowledge  includes  the  identity  of  all  objects  in  the  scene 
and  approximately  where  they  are;  the  goal  is  to  determine  the 
precise  location  of  one  or  more  of  the  objects. 

If  you  know  approximately  where  the  telephone 
is  (e.g.,  with  the  same  general  precision  as  Garvey's 
program)  and  you  wish  to  dial  a number,  you  would  verify 
that  the  phone  really  is  in  that  general  area  and  then 
locate  the  receiver  precisely  enough  to  pick  it  up  and 
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locate  the  holes  precisely  enough  to  insert  your  finger 
and  dial . 


VV  provides  a way  to  reduce  the  uncertainties  associated  with  the  location  of  an  object  It 
bridges  the  gap  between  the  known  tolerances  on  an  object  and  the  desired  tolerances  luhen 
the  initial  tolerances  greatly  restrict  the  possible  appearance  of  the  object.  VV  manipulates 
three-dimensional  object  models  and  it  can  determine  three-dimensional  corrections,  but  it 
assumes  that  the  two-dimensional  appearance  of  the  features  do  not  change  significantly  from 
one  execution  of  the  task  to  the  next  Thus,  VV  as  discussed  in  this  thesis  is  restricted  to 
quasi  two-dimensional  tasks  This  type  of  feedback,  however,  plays  an  important  role  in 
several  areas.  Programmable  assembly  is  one,  aerial  photograph  interpretation  and  medical 
image  processing  are  others 

VV  has  been  used  in  several  ways  in  the  past.  Possibly  the  best  known  is  within  the 
hypothesis  and  test  paradigm  For  example,  a high-level  procedure  hypothesizes  an  edge  at 
a certain  place,  the  verification  step  verifies  that  the  edge  is  there  and  computes  its  position 
and  angle  1 he  model  includes  exactly  what  will  appear  (the  edge),  approximately  where  (at 
some  position  and  orientation),  and  approximately  how  it  will  appear  (with  some  given 
contrast)  There  are  seveial  systems  in  which  this  type  of  VV  plays  a major  role  (see  [Falk 
70],  (Shirai  73],  [Tenenbaum  70],  and  [Gra|je  73]).  Another  area  in  which  VV  has  been  used 
is  narrow-angle  stereo  programs,  in  these  cases,  a model  is  a set  of  correlation  patches  from 
one  view  of  the  scene  and  the  goal  is  to  locate  these  patches  in  the  second  view.  Again  the 
model  states  the  identity  (the  unnamed  features  that  produce  the  correlation  patches),  the 
approximate  position  (near  the  back-projection  of  the  ray),  and  the  appearance  (a  slight 
vaiiation  from  the  correlation  patch).  See  [Quam  74],  [Hannah  74],  and  [Pingle  74]  for 
programs  of  this  'ype 

More  recently  there  has  been  considerable  interest  in  visual  perception  within  a 
programmable  assembly  system.  Such  systems  provide  complex,  but  predictable  environments. 
For  example,  consider  the  task  of  inserting  a screw  in  a hole  It  can  be  reduced  to  a few 
subtasks,  each  of  which  could  involve  VV: 

(1)  locate  the  hole  with  the  screw  outside  the  field  of 
view  ( see  f tgure  1.1), 

(2)  move  the  screwdriver  and  screw  into  the  picture  and 
locate  them  against  the  now  known  background  (see 
figure  1.2), 

(3)  decide  how  to  move  the  screw  closer  to  the  hole, 
and  (4)  return  to  step  2 if  necessary. 


Assume  that  a mechanical  arm  picks  up  the  part  with  the  hole  and  places  it  in  a vise  whose 
position  is  reasonably  well  known.  In  that  case  the  hole  may  appear  displaced  in  a picture  at 
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step  (I)  because  (a)  the  par:  does  not  seat  in  the  vise  exactly  as  planned  and  (b)  the 
calibration  between  the  arm  and  the  camera  is  not  exact  Finding  the  hole  m step  (1)  reduces 
these  factors  Thus,  there  are  fewer  uncertainties  for  step  (2)  For  step  (3)  the  main  factor 
contributing  to  the  error  will  be  the  imprecision  of  the  arm  since  the  problem  will  have  been 
reduced  to  an  analysis  of  the  relative  displacement  bet'-  een  the  tip  of  the  screw  and  the  hole 

More  and  more  information  about  the  expected  appearance  of  the  objects  can  be 
brought  to  bear  as  the  program  progresses  from  step  to  step  For  step  (1)  the  only 
information  may  be  a comparison  picture  of  this  same  step  during  a previous  assembly  and 
possibly  a synthetic  picture  generated  from  the  model  of  the  expected  scene  For  step  (2)  the 
picture  taken  at  step  (1)  is  available.  It  contains  the  background  that  will  appear  throughout 
the  task.  For  step  (3i  the  eailier  pictures  provide  information  ^oout  actual  glares,  shadows, 
and  light  levels  as  the  screw  approaches  the  hole 

Thus,  the  three  steps  offer  successively  greater  constraints  on  the  positions  of  the 
objects  and  greater  knowledge  about  the  appearances  of  the  objects.  The  increasing  amount 
of  information  should  make  each  successive  step  easier  and  faster.  The  algorithms  employed 
by  VV  are  specifically  designed  to  take  advantage  of  this  type  of  information.  VV  tries  to 
determine  the  cheapest,  most  reliable  way  to  locate  an  object  within  a desired  precision. 

V V is  not  the  only  method  that  a programmable  assembly  system  can  use  to  improve  its 
estimate  of  the  location  of  an  object.  In  the  past,  the  most  common  method  has  been  to  grasp 
an  object  at  two  or  three  places  and  then  combine  the  resulting  position  information  into  an 
estimate  for  the  location  of  the  object  Each  grasp  is  equivalent  to  a visual  operator  in  the 
information  that  it  gathers:  a value  (the  thickness  of  the  object  at  the  grasping  point)  and  a 
position  (the  position  of  a point  on  a plane  parallel  to  the  plane  of  the  fingers).  But  visual 
operators  hold  several  advantages  over  grasping  operators: 

(1)  Visual  operators  are  potentially  faster;  they  function  at  electronic 
speeds  as  opposed  to  the  mechanical  speeds  that  limit  touch  and 
force  feedback. 

(2)  Visual  operators  are  passive;  they  gather  information  about  an 
object  without  disturbing  it  This  may  be  important  for  small, 
delicate  parts. 

(3)  Visual  operators  offer  a wider  variety;  some  locate  a corner,  some 
locate  a point  on  a line,  some  locate  a point  on  a plane,  etc. 

(4)  Visual  operators  offer  a wider  range  of  scales;  the  same  operators 
can  be  used  with  microscopes  or  telescopes. 


(^)  Visual  fci'ctback  offers  a more  global  view  of  a situation.  For 
example,  it  is  virtually  impossible  to  use  force  feedback  to  decide 
which  way  to  proceed  after  a screw  has  missed  a hole.  A spiral 
search  by  the  arm  is  possible,  but  slow. 

(6)  Vi.nial  information  processing  can  often  be  performed  concurrently 
with  mechanical  motion.  For  example,  if  the  screwdriver  almost 
always  succeeds  in  picking  up  a screw  from  the  dispenser,  it  may  be 
possible  to  take  a picture  of  the  end  of  the  screwdriver  as  it  is 
leaving  the  dispenser  in  order  to  verify  that  the  screw  is  present. 

1 he  arm  can  move  toward  the  hole  while  the  VV  system  decides 
whether  or  not  the  screw  is  present.  If  the  system  decides  that  it  is 
present,  the  arm  is  free  to  continue  along  its  path.  However,  if  the 
screw  is  not  there,  the  VV  system  can  signal  the  arm  to  return  to  the 
dispenser  to  try  again.  The  economics  of  this  parallel  checking 
depends  upon  the  frequency  that  the  screw  is  missed  and  whether  or 
not  the  resources  are  otherwise  idle, 

1 his  list  of  advantages  should  not  be  taken  as  an  argument  for  the  exclusive  use  of  visual 
feedback  In  fact,  vision  is  most  effective  when  it  is  used  in  conjunction  with  touch  and  force 
sensing,  the  different  systems  can  check  each  other.  For  example,  if  visual  feedback  is  trying 
to  servo  a screw  into  a hole,  force  feedback  can  indicate  that  the  screw  has  missed  the  hole. 

One  long-range  goal  of  the  research  described  in  this  thesis  is  to  make  it  easier  for 
programmers  who  are  not  experts  in  computer  vision  to  program  VV  feedback.  In  the  past, 
simple  touch  and  force  feedback  have  been  understood  and  incorporated  into  mechanical  arm 
programs,  but  vision  has  been  treated  like  a never-never  land  by  anyone  not  involved  in 
vision  research  The  goal  is  to  make  vision  a viable  alternative  within  the  feedback  trio  of 
touch,  force,  and  vision.  To  do  that  requires  a way  to  extract  useful  information  from  a 
picture  of  the  workstation,  to  combine  the  results  of  such  extractions,  and  finally  to  make  a 
decision  based  upon  the  summarized  results  (see  figure  1.3). 

Programmers  generally  know  how  to  express  their  vision  tasks  in  terms  of  the  following 
three  quantities: 

(a)  the  confidence  that  the  system  has  found  the  correct  object(s), 

(b)  the  precision  within  which  the  system  has  located  the  object(s), 
and  (c)  the  cost  involved  in  determining  this  information. 

These  concepts  have  to  be  formalized  in  order  for  a programmer  to  specify  the  goal  of  a task 
precisely.  But  the  more  difficult  problem  is  to  develop  methods  that  produce  this  information 
from  pictures  of  the  scene.  There  are  two  aspects  of  this  second  problem:  (I)  the  extraction 
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of  individual  pieces  of  useful  inforniaiion  from  a picture  and  (2)  the  combination  of  several 
pieces  of  information  to  form  estimates  of  the  quantities  of  interest.  This  thesis  concentrates 
on  the  latter  problem 

The  implementation  of  the  VV  system  discussed  in  this  thesis  gathers  information  by 
applying  well-known  operaiots,  such  as  edge  operators,  correlation  operators,  and  region 
growers,  that  arc  designed  to  locate  and  describe  /((Hurts,  such  as  line  segments,  correlation 
points,  and  regions.  The  information  produced  by  such  operators  can  be  roughly  classified 
into  two  types:  value  information  and  position  information.  Value  information  includes  the 
value  of  a correlation  coefficient,  the  contrast  across  an  edge,  and  the  intensity  of  a region. 
Position  information,  in  addition  to  (x.y)  or  (x,y,z)  information,  may  include  orientation 
information  For  example,  an  edge  operator  might  return  the  (x,y)  position  of  a point  on  a 
line  and  an  estimate  of  the  oiientation  of  the  line.  This  information  is  classified  as  position 
information  The  same  edge  operator  may  compute  the  contrast  across  the  edge  and  the 
confidence  that  there  really  is  an  edge  at  that  position,  both  of  which  are  classified  as  value 
information 

The  distinction  between  value  information  and  position  information  is  natural  because 
often  it  is  reasonable  to  assume  that  values  from  different  operators  are  independent,  but  it  is 
seldom  reasonable  to  assume  that  positions  of  features  are  independent  (especially  features  of 
rigid  objects).  Independence  means  that  the  value  of  one  operator  (such  as  a correlator)  does 
not  affect  the  expected  value  for  another  operator  (such  as  an  edge  operator).  The  position 
information,  on  the  other  hand,  is  not  independent  because  the  location  of  one  point  or  the 
orientation  of  one  line  greatly  influences  the  possible  positions  for  other  features. 

Given  the  position  and  value  information  from  several  operators,  what  is  the  best 
estimate  of  the  location  of  an  object?  What  is  the  precision  associated  with  that  estimate? 
What  should  the  combination  rules  be?  In  the  past  the  combination  rules  have  been  designed 
for  specific  operators  and/or  tasks.  There  have  been  a few  special-purpose  programs  written 
that  perform  VV  tasks  within  programmable  assembly  environments  (e.g.,  see  [Holies  73]  and 
[Dewar  73))  and  a few  programs  that  handle  a subclass  of  the  VV  tasks  (e.g.,  see  [Agin  75], 
[Fischler  7 lb],  [Chien  75]  and  [Holland  75]).  but  none  of  these  programs  concentrates  on 
precision  and  confidence  and  is  general  enough  to  accomplish  a wide  variety  of  VV  tasks. 

Part  of  this  thesis  describes  a set  of  mathematical  tools  that  form  a set  of  combination 
rules  for  the  class  of  VV  tasks.  A least-squares  technique  is  used  to  combine  available 
position  information  to  form  a current,  best  estimate  of  the  location  of  the  object  (plus  a 
tolerance  about  that  estimate).  Bayesian  probability  is  used  within  a sequential  decision 
scheme  to  compute  the  necessary  confidences.  These  techniques  are  well-known,  but  they 
combine  paiticularly  nicely  to  answer  the  various  needs  of  a VV  system. 


Input  the  initial  constraints 
on  the  objects'  positions 


Any  maximum  cost 
limits  exceeded? 


RETURN  FAILURE 


NO  Are  there  more 
untried  operators? 


Choose  the  next  operator 


Determine  the  region  in  uhich 
the  feature  might  appear 


Apply  the  operator 


Compute  the  best  estimate  and  precision 
for  the  objects*  positions 


Compute  the  confidence 


RETURN  SUCCESS 


Are  the  confidence  and  precision  high  enough? 


Figure  1.4 
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Figure  I “1  shows  the  general  flow  of  control  for  a VV  program  based  upon  these  ideas. 
The  flowchart  describes  the  subtasks  performed  at  execution  time  There  are  also  several 
preliminary  subtasks  that  have  to  be  penormed  in  order  to  produce  such  a program.  The 
following  IS  a list  of  some  of  the  subtasks  involved  in  VV: 

(1)  Given  a specific  set  of  objects,  suggest  some  candidate  features  and 
operators  to  find  such  features. 

(2)  Determine  the  infoimation  that  a specific  operator  can  contribute 
toward  the  confidence  that  the  correct  object  has  been  found. 

(3)  F.stimate  the  expected  cost  of  applying  operator  X. 

(4)  Determine  the  actual  cost  of  applying  operator  X. 

(5)  Select  the  next  operator  to  be  applied. 

(6)  Combine  the  results  of  several  operators  to  give  an  overall 
confidence. 

(7)  Estimate  the  location  of  an  object  based  upon  the  results  of  several 
operators. 

(8)  Predict  the  expected  number  of  operators  required  to  achieve  a 
certain  confidence 

These  subtasks  can  be  partitioned  according  to  the  time  at  which  they  are  most  important. 
For  example,  to  predict  the  expected  number  of  operators  is  important  at  planning  time  when 
the  program  or  user  is  trying  to  decide  the  expected  cost  of  accomplishing  the  task. 
Suggesting  candidate  features  is  important  at  programming  time  when  the  user  is  describing 
potential  sources  of  information.  Four  such  times  or  stages  will  be  distinguished  within  this 
thesis: 

(1)  PROGRAMMING  TIME:  the  user  states  the  goal  of  the  task, 
specifics  the  confidence,  precision,  and  cost  constraints,  and 
interactively  chooses  potential  features  and  operators. 

(2)  TRAINING  TIME;  the  program  applies  the  chosen  operators  to 
several  sample  pictures  and  gathers  statistical  information  about 
their  effectiveness. 
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(3)  PI  ANNJNC  TIME,  the  system  ranks  the  operators  according  to 
their  expected  contribution,  determines  the  expected  number  of 
operators  to  be  needed,  and  predicts  the  cost  of  accomplishing  the 
task. 

(•1)  EXECUTION  TIME:  the  system  applies  operators  one  at  a time, 
combines  the  results  into  confidences  and  precision,  and  stops  when 
the  desired  levels  have  been  reached  or  a cost  limit  has  been 
exceeded. 

To  accomplish  a VV  task  requires  progress  from  one  stage  to  the  next  in  the  order 
shown  above.  However,  for  clarity,  these  stages  will  be  discussed  in  a different  order 
execution  time,  planning  time,  programming  time,  and  training  time.  The  execution-time 
discussion  describes  how  the  results  of  operators  are  combined  for  the  two  types  of  tasks, 
inspection  and  location;  the  planning-time  discussion  describes  how  the  application  order  and 
number  of  the  operators  is  determined;  the  programming-time  discussion  describes  how  an 
operaior/featurc  pair  is  chosen  as  a potential  source  of  information;  the  training-time 
discussion  describes  how  the  potential  benefits  of  an  operator  are  characterized  from  several 
trial  runs.  1 he  thesis  concludes  with  a description  of  some  possible  extensions. 

This  thesis  relics  heavily  upon  the  domain  of  programmable  assembly  for  its  examples 
and  motivation  (eg.,  see  [Finkel  75]  and  {Finkel  76]).  The  techniques  are  discussed  in  the 
context  of  a highly  controlled  environment  in  which  mechanical  arms  are  performing 
assembly  task's.  Some  of  the  techniques  have  been  optimized  to  take  advantage  of  specific 
properties  of  this  environment,  but  the  basic  methods  used  to  produce  location  and  confidence 
information  from  the  results  of  several  visual  operators  are  more  widely  applicable  Other 
possible  areas  include  photo-interpretation  and  medical  image  processing. 

1 here  is  one’ other  general  remark  that  should  be  made  at  the  beginning  of  this 
discussion.  Although  most  of  the  examples  and  techniques  described  here  are  based  upon 
conventional  television  cameras  and  their  images,  there  is  no  reason  why  the  same  or  similar 
techniques  could  not  be  used  within  systems  based  upon  direct  ranging  devices,  laser  trackers, 
or  multiple  touch  sensors. 
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CHAPTER  2 
MOTIVATION 


The  purpose  of  this  chapter  is  to  outline  the  general  requirements  and  the  semantic 
structures  necessary  for  a VV  system  that  can  be  easily  programmed  (see  [Bolles  75]).  Two 
example  tasks  and  their  solutions  are  used  to  introduce  the  desired  capabilities.  Later 
chap^rs  will  discuss  specific  mechanisms  in  detail.  Appendices  IV  and  V contain  traces  of  a 
programmer  using  the  current  implementation  of  the  VV  system  to  set  up  and  test  VV 
programs  that  perform  the  two  tasks  discussed  in  this  chapter. 

I The  two  example  tasks  are  (1)  check  for  a screw  on  the  end  of  the  screwdriver  and  (2) 

locate  a sciew  hole  in  a part  that  has  been  placed  in  a vise.  The  goal  of  the  first  task  is  to 
make  a yes  or  no  decision:  Is  the  screw  present?  VV  tasks  of  this  type  will  be  referred  to  as 
inspection  tasks.  The  goal  of  a location  task,  on  the  other  hand,  is  to  estimate  the  current 
IcKation  of  the  object,  or  equivalently  to  estimate  the  displacement  between  the  current 
Icxation  of  the  object  and  its  planned  location.  The  success  of  an  inspection  task  is  usually 
measured  by  its  confidence-,  the  success  of  a location  task  is  usually  measured  by  its  precision. 


Section  1 

CHECKING  FOR  A SCREW 


Consider  visually  deciding  whether  ' or  not  there  is  a screw  on  the  end  of  the 
screwdriver.  One  idea  for  a solution  is  to  aim  a camera  at  the  exit  of  the  screw  dispenser, 
take  a picture  of  the  end  of  the  screwdriver  as  it  leaves  the  dispenser,  apply  a series  of 
operators  to  the  picture,  and  use  the  values  of  the  operators  to  decide  whether  or  not  the 
screw  is  on  the  end.  There  are  several  decisions  to  be  made  before  this  idea  can  be  converted 
into  an  algorithm  to  perform  the  desired  task;  Which  visual  operators  should  be  applied? 
Where  should  they  be  applied?  In  what  order  should  the  operators  be  applied?  How  should 
the  results  of  several  operators  be  combined?  How  much  can  an  operator  contribute  toward 
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the  final  decision?  These  and  other  questions  will  be  briefly  discussed  below. 


/.  What  are  some  potentially  useful  operatorl feature  pairs? 

Since  the  goal  of  this  task  is  to  decide  whether  or  not  the  screw  is  on  the  end  of  the 
screwdriver,  each  operator  chosen  should  contribute  toward  this  decision.  Consider  figure 
2.1.1,  which  shows  typical  pictures  of  the  two  expected  situations:  one  with  the  screw  on  the 
end  of  the  screwdriver  and  one  with  the  screw  missing.  An  operator  that  can  distinguish  the 
texture  formed  by  the  screw  threads  is  a potentially  useful  operator,  because  the  presence  of 
screw  threads  distinguishes  the  two  situations.  An  operator  that  locates  a feature  on  the 
screwdriver  would  not  directly  contribute  to  a decision  because  the  screwdriver  is  present 
whether  the  screw  is  there  or  not.  But  this  operator  may  still  be  useful  to  improve  the 
system’s  estimate  of  the  location  of  the  screwdriver  tip  and  hence  make  it  easier  for  the 
texture  operator  to  find  the  screw  threads.  Thus,  operators  may  directly  or  indirectly 
contribute  to  the  final  decision. 


Figure  2.1.1 


In  general,  the  objects  in  a scene  have  several  visual  features:  colors,  textures,  sizes, 
edges,  corners,  and  holes.  The  more  unique  features  an  object  has,  the  easier  it  is  to  find. 
One  reason  the  blocks  world  is  difficult  to  analyze  visually  is  that  blocks  do  not  have  very 
many  distinctive  features.  All  blocks  have  edges,  corners,  and  planes  so  it  is  difficult  to 
distinguish  one  from  another.  Programmable  assembly,  on  the  other  hand,  offers  a wide 
variety  of  features. 
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Given  a wide  variety  of  features,  which  ories  would  be  useful  to  locate?  Which 
operators  should  be  applieci  to  find  such  features?  A VV'  program  can  provide  a user  with 
three  levels  of  assistance  in  order  to  help  him  choose  potentially  useful  operator/feature  pairs 
for  a specific  task: 

(1)  The  VV  system  can  provide  a convenient  environment  within 
which  to  experiment  with  different  operators.  The  user  might  describe 
features  such  as  lines,  corners,  and  correlation  patches  and  apply  various 
operators  to  locate  them  in  trial  pictures.  For  example,  the  user  could 
describe  the  corner  formed  by  the  head  of  the  screw  and  the  shaft  of  the 
screwdriver  (see  figure  21.2),  try  a corner-finder  to  find  it,  try  a 
correlation  operator  to  find  it,  choose  the  more  promising  operator,  and 
add  the  chosen  operator  and  corner  to  the  list  of  potentially  useful 
operator/feature  pairs. 

(2)  The  VV  system  can  analyze  a typical  picture  of  the  scene  to 
pioduce  a ranked  list  of  potential  operator/feature  pairs.  For  example, 
an  edge  operator  may  be  applied  to  the  picture  in  order  to  pick  out  all 
pairs  of  line  segments  that  form  a corner  of  a certain  minimum  size. 

One  of  these  corners  might  be  the  corner  formed  by  the  head  of  the 
screw  and  the  shaft  of  the  screwdriver.  Since  both  a corner-finder  and  a 
correlation  operator  can  locate  corners,  two  operator/feature  pairs  can  be 
added  to  the  list  of  suggestions  for  each  corner;  a corner-finder/corncr 
pair  and  a correlation  operator/corner  pair. 

This  type  of  automatic  operator/feature  suggestion  procedure 
reduces  the  amount  of  detailed  work  required  of  the  user.  The  user  only 
has  to  filter  out  suggestions  that  are  difficult  to  locate  reliably  or  that 
produce  unreliable  position  information. 

(3)  The  VV  system  can  analyze  a model  of  the  scene  to  produce  a 
ranked  list  of  operator/feature  pairs.  The  more  complete  the  model,  the 
better  the  suggestions.  The  model  might  include  a three-dimensional 
representation  of  all  the  objects  in  the  scene,  a model  of  the  camera,  and 
a model  of  the  light  sources.  For  example,  a hidden-line  elimination 
scheme  can  be  used  to  predict  visible  corners  from  models  of  the  screw 
and  Krewdriver.  One  of  these  corners  might  be  the  one  formed  by  the 
head  of  the  screw  and  the  shaft  of  the  screwdriver. 

This  type  of  automatic  suggestion  procedure  can  analyze  potential 
operator/feature  pairs  at  a higher  level  than  the  system  mentioned  above. 
Potentially  it  can  bring  to  bear  all  the  knowledge  associated  with  real 
objects:  their  appearance,  their  structure,  and  their  function.  However, 
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the  icsnlts  31  c only  as  good  as  the  model,  whereas  in  (2)  the  results  are 
only  as  good  as  the  training  picture. 

This  list  is  ordered  according  to  the  sophistication  of  the  proposed  system;  the  later 
procedures  require  more  information  about  the  task  and  they  can  analyze  the  potential 
suggestions  more  efficiently  For  example,  the  second  level  might  suggest  a corner  feature,  one 
side  of  which  is  formed  by  a shadow.  If  the  shadow  changes  from  one  trial  picture  to  the 
next,  the  feature  is  probably  not  a good  one.  The  only  way  that  the  second  level  system  can 
arrive  at  this  conclusion  is  (1)  to  monitor  the  relative  positions  of  several  features  in  several 
trial  pictures  and  (2)  to  notice  that  the  corner  changes  position  with  respect  to  the  other 
features  The  third  level  system  could  directly  determine  that  one  side  is  formed  by  a shadow 
and  discard  this  feature  immediately. 


2.  Where  should  an  operator  be  applied^ 

In  the  scicw  in.sppction  task,  since  the  arm  and  camera  are  not  exact,  the  end  of  the 
screwdriver  will  sometimes  appear  at  one  point  in  the  picture  and  sometimes  at  another.  If 
the  total  range  of  possihic  positions  is  only  a small  portion  of  the  picture,  there  is  no  reason  to 
apply  operatois  over  the  whole  picture.  The  region  of  possible  positions  for  a feature  in  a 
picture  will  be  refcired  to  as  the  toleiance  region  for  the  feature.  It  can  be  determined  once 

during  a programming  session  and  can  remain  fixed  during  the  training,  planning,  and 

execution  phases  In  order  to  find  the  feature  during  one  of  these  phases,  only  its  tolerance 
region  must  be  scanned,  not  the  whole  picture. 

How  can  the  tolerance  region  for  a feature  be  determined?  One  way  would  be  to  have 
the  arm  attempt  to  get  a screw  from  the  dispenser  several  hundred  times,  take  a picture  after 
each  attempt,  mark  the  position  of  the  end  of  the  screwdriver  in  each  of  the  pictures, 
incorporate  all  of  the  occurrences  into  a continuous  region  (eg.,  a convex  polygon),  and 
declare  that  region  to  be  the  tolerance  region  for  the  end  of  the  screwdriver. 

Another  way  to  determine  the  tolerance  region  about  a feature  is  to  ask  the  user  to 
specify  the  constraints  that  limit  the  uncertainty  associated  with  the  position  of  the  feature. 
For  the  scicw-chccking  example  tl.e  ronstraints  would  include  such  information  as  the 
accuracy  of  the  arm  and  the  accuracy  of  the  hand  grasping  the  screwdriver.  These 
constraints  can  be  translated  into  a three-dimensional  volume  that  represents  the  possible 
positions  of  the  feature.  This  volume  will  be  referred  to  as  the  tolerance  volume  of  the 

feature  The  toleiance  region  for  the  feature  can  be  formed  by  projecting  its  tolerance 

volume  onto  the  screen. 

In  order  to  project  the  three-dimensional  volume  onto  the  camera  screen  a camera 
calibration  is  needed  Camera  calibrations  are  designed  to  provide  (I)  a transform  that  maps 


a point  in  the  workstation  coordinate  system  onto  a point  in  the  screen  coordinate  system  and 
(2)  a transform  that  maps  a point  in  the  screen  coordinate  system  into  a ray  in  the  workstation 
ccxirdinate  system.  Some  additional  piece  of  information  (eg.,  the  height  of  a point)  is 
necessary  to  map  a point  on  the  screen  into  a point  in  the  workstation  coordinate  system  (eg., 
see  (Sobcl  74]).  Camera  calibrations  are  not  exact,  so  tolerance  regions  should  be  expanded  to 
accommodate  for  this  additional  inaccuracy.  The  application  of  a camera  calibration  to  move 
back  and  forth  between  the  screen  coordinate  system  and  the  workstation  coordinate  system 
will  be  used  several  times  throughout  the  remainder  of  this  discussion. 

Given  a feature,  its  tolerance  region,  and  an  operator  to  find  the  feature,  where  should 
the  operator  be  applied  within  the  region  to  locate  the  best  match  for  the  feature?  The 
search  strategy  depends  upon  several  factors,  including  the  type  of  feature,  the  operator  being 
used,  the  size  of  the  tolerance  region,  and  the  feature's  expected  distribution  of  positions 
within  the  tolerance  region  If  a correlation  operator  is  used  to  locate  the  corner  formed  by 
the  head  of  the  sciew  and  the  shaft  of  the  screwdriver,  an  exhaustive  search  may  be 
reasonable  if  the  tolerance  region  is  small.  When  an  edge  operator  is  used  to  locate  a point 
on  a line  segment,  a few  linear  scans  across  the  tolerance  region  are  often  sufficient.  A set  of 
starch  techniques  and  met.  xls  to  predict  their  expected  cost  in  different  situations  are 
required  in  order  to  choose  a good  search  strategy. 


How  should  the  results  of  an  operator  be  interpreted? 

Consider  a hypothetical  texture  operator  that  ranks  local  regions  in  a picture  according 
to  their  similarity  to  the  texture  formed  by  the  screw  threads.  It  may  return  a value  of  .95 
when  applied  to  an  example  picture  with  the  screw  present  (see  figure  211)  and  .57  when 
applied  to  an  example  picture  with  the  screw  missing.  If  the  same  operator  is  applied  to  a 
new  picture  to  decide  whether  or  not  the  screw  is  present,  and  it  returns  a value  of  .86,  what 
IS  the  probability  that  the  screw  is  on  the  «nd  of  the  screwdriver?  How  can  the  a priori 
probabilities  be  incorporated  into  the  computation? 

If  one  operator  implies  that  there  is  a probability  of  .76  that  the  screw  is  present  and 
another  operator  implies  that  there  is  a probability  of  .84  that  the  screw  is  present,  what  is  the 
overall  probability  of  the  screw  being  present?  How  are  the  results  of  the  two  operators 
related?  In  general,  how  can  the  values  of  several  operators  be  integrated  into  one  estimate 
for  the  probability  that  the  screw  is  present? 

There  are  two  errors  that  can  be  made  in  a task  of  this  sort:  (a)  the  operator  values 
may  imply  that  the  screw  is  present  when  it  is  not,  and  (b)  the  operator  values  may  imply  that 
the  Krew  is  missing  when,  in  fact,  it  is  present.  These  errors  are  referred  to  as  errors  of  the 
first  and  second  type,  respectively  [Shewhart  39].  How  can  an  assembly  engineer  set  the  limits 
on  the  acceptable  number  of  errors  of  each  type?  Given  such  limits  and  a set  of  operators. 
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what  IS  the  e,x.f)ccted  number  of  o|jeiatois  that  will  have  to  be  applied  in  order  to  make  a 
decision  that  satisfies  these  limits? 

The  combination  rules  are  complicated  by  the  fact  that  visual  operators  are  not 
completely  reliable.  Sometimes  the  best  match  tficy  find  is  not  the  intended  match.  For 
example,  consider  the  coriel.ition  ojicraior  shown  m figure  2.1  3.  If  it  is  applied  throughout 
the  tolerance  region  shown  in  figure  2 I 3.b,  it  will  piobably  find  two  good  matches,  as  shown 
in  figure  2. 1.3.c.  If  the  operator  happens  to  prefer  match  it  will  return  an  incorrect  match. 
In  general,  when  an  operator  is  applied  at  several  positions  within  a tolerance  region,  it 
returns  a set  of  different  values.  Which  position  is  the  best  match?  What  is  the  chance  that 
the  best  match  is  really  the  correct  match?  How  can  the  combination  rules  adjust  for  this 
unreliability? 

T he  possibility  of  unreliable  operators  means  that  (1)  the  user  should  check  for 
potential  cotifusiotis  at  programming  tinie;  (2)  the  training-time  subsystem  should  gather 
statistics  on  the  reliability  of  the  operators;  (3)  the  planning-time  subsystem  should  reduce  the 
desirability  of  unreliable  operators;  and  (-1)  the  execution-time  combination  rules  should 
reduce  the  contribution  of  any  operator  known  to  be  unreliable. 


4.  Which  operator  shouhl  be  applied  first? 

Some  operators  find  their  matches  more  easily  than  others;  some  operators  contribute 
more  toward  the  final  decision  than  others.  In  what  order  should  the  operators  be  applied? 
For  example,  in  the  screw-checking  task  the  screw  thread  operator  may  be  faster  than  the 
corner-finder,  but  the  corner-finder  may  produce  more  information  than  the  screw  thread 
operator.  Which  one  should  be  applied  first? 

One  possible  choice  mechanism  is  to  apply  the  operator  with  the  largest  expected  value 
for  the  ratio 

<contribution  toward  the  final  decision) 


<cost> 

first.  But  what  is  the  expected  contribution  of  an  operator?  At  execution  time  the 
combination  rules  compute  a specific  contribution  for  a specific  value  of  an  operator.  The 
planning  mechanism  needs  the  average  contribution  of  the  operator,  which  depends  upon  the 
distribution  of  the  values  of  the  operator  as  well  as  the  contribution  derived  from  each  value 

As  mentioned  earlier,  operators  can  contribute  indirectly  as  well  as  directly  toward  the 
final  decision.  How  can  these  indirect  contributions  be  incorporated  into  the  ranking  of  the 
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operators?  Koi  example,  an  edge  operator  that  locates  a point  on  the  side  of  the  shaft  of  the 
sctewdrivei  may  be  the  cheapest  operator  to  apply  in  the  screw-checking  task.  It  only 
indirectly  contributes  to  the  task  by  increasing  the  constraints  on  the  position  of  the 
screwdiiver  and  sciew,  but  it  still  may  be  the  best  first  step  toward  the  final  decision.  How 
can  the  strategist  take  this  type  of  expected  progress  into  account? 


5.  A rt-JinCii  PK  program 

In  light  of  this  discussion  the  executibn-time  program  for  the  screw-checking  task  can 
be  restated  as  follows 

(a)  Aim  the  camera  at  the  exit  of  the  screw  dispenser. 

(b)  Calibrate  the  camera. 

(c)  Take  a picture  of  the  end  of  the  screwdriver  as  it  leaves  the 
dispenser. 

(d)  Apply  one  operator  at  a time,  using  the  best  one  first.  For  each  one, 
employ  the  appropriate  search  technique  to  locate  the  best  match 
w'itliin  the  tolerance  region  of  the  feature. 

(e)  Incrementally  incorporate  the  results  of  each  operator  into  an 
estimate  of  the  probability  that  the  screw  is  present.  Be  aware  of 
possible  confusions  and  adjust  the  probabilities  accordingly. 

(f)  Strip  ap|ilying  operators  and  make  a decision  as  soon  as  one  can  be 
m.'ide  with  the  desired  confidence. 

The  calibration  steps  (steps  a and  b)  may  be  performed  only  once  for  each  series  of 
assemblies.  T he  remaining  steps  are  performed  each  time  the  arm  tries  to  obtain  a screw 
from  the  screw  dispenser. 
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Section  2 

LOCATING  A HOLE 

Consider  visually  locating  a screw  hole  in  a part  that  has  been  placed  in  a vise.  Such  a 
task  is  a location  task;  the  user  is  more  concerned  about  the  location  and  precision  associated 
with  the  hole  than  about  the  confidence  that  the  hole  is  present.  There  is  no  question  about 
what  is  sought,  just  an  uncertainty  about  uihert  it  is. 


I 

m 

■ 

■ 

s 

I 
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Figure  2 2.1 

Assume  that  a flat  side  of  the  part  is  placed  in  contact  with  one  of  the  vise  Jaws  (see 
figure  22.1).  Then  the  part  can  only  be  rotated  and  translated  in  the  plane  parallel  to  the 
Jaws.  If  the  vise  location  is  well-known,  the  uncertainties  associated  with  the  location  of  the 
part  can  be  described  u a planar  transformation,  which  is  a function  of  three  parameters:  dx. 
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<{y,  and  d--  The  |iaiametcis  dx  and  dy  ate  unknown  translations  in  the  plane  of  the  vise 
jaws;  dr.<  is  an  unknown  rotation  about  a vector  jjci  pendicular  to  the  plane  of  the  vise  jaws. 
Constraint  tesolving  routines  can  combine  the  accuracy  of  the  arm  and  the  accuracy  of 
grasping  the  part  to  produce  a prioii  limits  on  these  parameters.  For  example. 

-1.32  cm  i dx  S 1.32  cm 
-.76  cm  < dy  S .70  cm 
-10.4  degrees  s dty  s 10.4  degrees. 

1 he  goal  of  the  task  is  to  improve  these  limits  to  a prescribed  value.  For  example,  the  arm 
control  progiam  might  need  to  know  the  location  of  the  screw  hole  to  within  the  following 
tolerances: 


-.15  cm  i dx  S .15  cm 

- . 1 5 cm  5 rfy  ^ . 1 5 cm. 

A general  way  to  accomplish  this  goal  is  to  apply  several  operators  and  combine  their 
position  information  to  form  a better  estimate  for  the  location  of  the  screw  hole  and  a 
measuie  of  the  precision  of  that  estimate.  What  are  the  combination  rules  necessary  to 
produce  such  estimates,  and  how  can  the  pircision  of  each  individual  operator  be  taken  into 
account’  These  and  other  questions  will  be  briefly  discussed  in  the  following  subsections. 


I How  can  position  estimates  and  precisions  be  computed^ 

In  the  hole-locating  task  each  operator  locates  a specific  point  on  the  part  in  the  vise 
The  function  of  the  combination  rules  is  to  determine  the  values  of  dx,  dy,  and  du.  that 
transfoim  the  expected  (oi  planned)  jjositions  of  the  features  into  the  observed  positions  of 
the  features  1 his  sei  of  combination  rules  will  be  referred  to  as  the  fitting  scheme 

Fitting  schemes  encounter  several  complicating  factors.  First,  visual  operators  locate 
matches  in  a picture  of  a scene  and  hence  pioduce  position  information  within  the  coorainate 
system  of  the  camera  screen,  not  the  workstation.  Since  the  arm  control  program  nteds  to 
know  the  location  of  the  hole  in  the  workstation  coordinate  system,  the  position  information 
has  to  be  transformed  by  the  camera  calibration  into  the  workstation  coordinate  system.  Jf 
two  or  moie  cameias  aie  being  used,  stereo  techniques  can  directly  locate  features  in  the 
woikstation  coordinate  system  With  one  camera,  however,  additional  position  information  is 
needed  to  deiermirie  a feature’s  woikstation  coordinates.  In  the  screw  hole  task,  since  the 
uncertainties  have  the  form  of  a planar  transform,  the  distance  from  a feature  to  the  known 
pl.me  is  sufficient.  1 hat  is,  given  the  screen  coordinates  for  the  image  of  a feature  and  the 
height  of  the  feature,  it  is  possible  to  locate  the  unique,  three-dimensional  position  for  the 
feature.  Stereo  is  preferable,  but  useful  information  can  be  derived  from  a single  camera  and 
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some  additional  information. 

The  second  complication  is  that  some  operators  are  more  precise  than  others.  For 
example,  a correlation  operator  may  locate  the  corner  formed  by  the  head  of  the  screw  and 
shaft  of  the  screwdriver  to  within  1.5  pixels  and  the  corner-finder  may  lixate  the  same  corner 
to  within  half  a pixel.  (A  pixel  is  one  resolution  element  in  a digitized  picture.)  The  fitting 
scheme  should  be  able  to  weight  the  results  of  these  operators  appropriately. 

A third  complication  is  that  some  operators  do  not  locate  a specific  point  on  the  part; 
they  locate  a piece  of  a line  or  a portion  of  a region.  For  example,  the  edge  operator  that 
looks  for  the  side  of  the  shaft  locates  a point  on  a line  segment  and  the  hypothetical  screw 
thread  operator  locates  a portion  of  a small  region.  The  fitting  scheme  should  be  general 
enough  to  incorporate  these  different  types  of  position  information. 

A fourth  complication  is  that  an  operator  may  locate  a decoy  instead  of  the  correct 
match.  If  the  decoy  looks  locally  like  the  desired  match,  the  only  way  to  determine  that  it  is  a 
decoy  is  to  check  the  global  structure  of  the  matches.  Since  the  features  of  a rigid  object  are 
expected  to  remain  at  fixed  relative  positions,  it  should  be  possible  to  check  a set  of  feature 
matches  for  their  structural  consistency.  Do  the  matching  positions  correspond  to  some 
reasonable  transformation  of  the  object?  Structural  consistency  can  also  be  incorporated  into 
the  confidence  computations  used  in  inspection  tasks.  If  the  program  is  checking  for  a 
vertical  screw,  and  a pair  of  operators  imply  that  the  tip  of  the  screw  and  the  top  of  the  screw 
are  side  by  side,  one  ought  to  be  suspicious. 

‘ This  discussion  of  decoys  and  unreliable  operators  suggests  two  general  conclusions:  (1) 

the  confidence  that  an  operator  has  located  the  correct  match  is  just  as  important  within 
' location  tasks  as  it  is  within  inspection  tasks,  and  (2)  position  information  (ie.,  structural 

i consistency)  can  make  a significant  contribution  in  an  inspection  task  toward  the  overall 

I confidence  in  a decision.  In  other  words  inspection  tasks  should  be  concerned  with  position 

I information  in  addition  to  confidences,  and  location  tasks  should  be  concerned  with 

I confidences  in  addition  to  position  information.  Thus,  structural  consistency  and  confidence 

I are  important  within  both  types  of  tasks. 

I Finally,  the  fitting  scheme  should  be  able  to  incorporate  new  information  incrementally. 

^ As  more  operators  are  applied  and  matches  are  found,  their  results  should  be  combined  with 

, previous  results.  Intermediate  estimates  can  be  used  to  reduce  the  amount  of  searching 

I required  to  locate  a new  feature.  For  example,  after  a point  on  the  shaft  of  the  screwdriver 

has  been  located,  its  position  and  precision  can  be  combined  with  a priori  limits  associated 
with  the  location  of  the  screw  to  form  a new,  smaller  tolerance  region  about  the  corner  formed 
by  the  head  of  the  screw  and  the  side  of  the  shaft.  The  use  of  intermediate  estimates  to 
reduce  tolerance  regions  re-emphasizes  the  importance  of  the  order  of  the  operators  and  the 
’ need  for  a sequential  strategy  to  locate  the  desired  object  to  within  the  desired  precision. 
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2.  A Ttjintd  K/''  program  to  locate  the  screw  hole 
Thr  cxTcution-timc  VV  program  to  locate  the  screw  hole  can  be  restated  as  follows: 

(a)  Aim  the  camera  at  the  vise 

(b)  C alibi  ate  the  camera 

(c)  Apply  one  operator  at  a time  in  the  most  strategic  order. 

(d)  Use  earh  operator's  position  information  to  update  the  estimates  for 
the  parameters,  dx,  dy.  and  dec. 

(e)  Use  the  estimates  for  the  parameters  to  determine  the  expected 
position  of  the  next  feature  and  a new,  improved  tolerance  region 
about  that  position. 

(f)  r.e  cautious  about  the  implications  dra.vn  from  the  position 
infoimation  because  the  operators  are  not  completely  reliable. 


Section  3 

SUMMARY  OF  REQUIRED  FACILITIES 


This  section  summaiires  the  facilities  reejuired  by  a VV  system.  It  lists  the  main 
components,  states  their  role  within  the  complete  system,  and  briefly  describes  the  current 
$tate-of-lhe-ail  in  each  area 


I.  Calibration  Techniques 

Calibiation  loiitities  provide  transformations  back  and  forth  between  the  screen 
coordinate  system  and  the  workstation  coordinate  system.  Almost  every  computer  vision 
system  has  its  own  methods  for  calibrating  the  cameras.  The  basic  techniques  can  be  found 
in  [Sobel  74],  [Sproull  73],  and  [Raumgart  74b]. 

Calibiation  can  itself  be  described  as  a VV  task:  place  a known  object  at  a known 
position,  visually  locale  the  object,  and  compute  the  current  position  of  the  camera.  The 
advantage  of  this  formulation  is  that  the  same  VV  program  can  be  used  to  calibrate  the 
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cameras  and  to  perform  the  desired  feedback  task;  a special-purpose  calibration  subsystem  is 
not  necessary.  Another  advantage  of  this  approach  is  that  the  camera  can  directly  calibrate 
itself  with  respect  to  the  stationary  objects  in  the  scene  (eg  the  vise  or  the  screw  dispenser) 
instead  of  a special  calibration  object.  Gill  used  his  corner-finder  to  develop  a version  of  this 
type  of  calibration  system  for  the  blocks  world  (see  [Gill  72]). 

A calibration  routine  may  also  include  an  estimate  for  the  camera  noise  and  a 
description  of  the  lens  distortion.  These  quantities  are  important  factors  in  predicting  the 
appearances  and  positions  of  features.  Gennery  and  Moravec  have  developed  a program  to 
estimate  these  quantities  (Moravec  76]. 

2.  yisuat  Operators 

A visual  operator  locates  the  image  of  a feature  and  return*  an  estimate  for  its  location 
and  a description  of  its  appearance.  For  example,  a region  grower  might  locate  a small,  dark, 
elliptical  region  inside  a larger,  grey  region;  an  edge  operator  might  locate  a distinct, 
high-contrast  edge  with  some  particular  orientation  at  a certain  location. 

There  are  several  well-known  types  of  operators:  edge  operators  (eg.  see  [Roberts  63], 
(Horn  71],  and  [Hueckel  69)),  line  followers  (eg.  see  [Tenenbaum  70]  and  [Shirai  73]),  corner 
finders  (eg  see  [Gill  72]  and  [Perkins  73]),  correlation  operators  (eg.  see  [Quam  7l],  [Hannah 
71],  and  [Moravec  76)),  region  growers  (eg.  see  [Brice  70],  [Yakimovsky  73a],  [Agin  75],  and 
[Garvey  76]),  and  texture  operators  (eg.  sec  [Bajesy  72],  [Lieberman  7l]  and  [Marr  75a]). 
There  is,  however,  still  a need  for  a wider  variety  of  more  powerful  operators.  Some  of  the 
most  useful  would  be  operators  to  grow  textured  regions  and  locate  boundaries  between  two 
textured  regions. 


S.  Search  Strategies 

The  purpose  of  a search  strategy  is  to  choose  where  to  apply  an  operator  in  order  to 
locate  the  best  (or  at  least  a good)  match  as  cheaply  as  possible.  Strategies  may  include 
heuristics  to  avoid  the  cost  of  an  exhaustive  search  For  example,  Moravec  uses  a 
two-dimensional  binary  search  to  locate  a good  match  for  a correlation  operator  (see 
[Moravec  76]).  Exhaustive  searches  can  also  be  avoided  when  looking  for  extended  features, 
such  as  lines  and  regions.  The  larger  the  feature,  the  easier  it  is  to  find. 

If  an  operator  is  known  to  be  unreliable,  a search  strategy  may  be  used  to  produce  an 
ordered  list  of  the  best  three  or  four  matches  for  the  operator.  A VV  program  may  try  one 
match  after  another  until  it  finds  a structurally  consistent  one. 


‘f  A Light  Model 


A lioht  model  describes  the  position,  brightness,  and  spectrum  of  each  light  source  at 
the  worksf.ation  It  can  be  used  in  conjunciion  with  the  object  models  to  predict  the 
appearance  of  an  object  The  angle  of  incidence,  the  reflectance  of  the  object,  and  the  angle 
of  obsei  vation  are  the  basic  variables  that  determine  the  appearance  of  a point  on  an  object 

Theie  are  several  piograms  that  use  object  models  and  light  models  to  produce 
synthetic  pictures  borne  of  the  most  complete  graphic  display  systems  have  been  developed  at 
the  University  of  Utah  (eg.,  see  [Couraud  71]).  Such  systems  are  general  enough  to  handle 
objects  with  smooth,  curved  surfaces  (eg.  car  bodies),  transparent  objects  (eg.  wine  glasses), 
and  shadows.  1 he  two  main  restrictions  associated  with  these  programs  are  that  (1)  the 
object  models  arc  difficult  to  construct  and  (2)  they  require  a great  deal  of  computation  time 

Horn  at  MIT  [Horn  70]  has  investigated  techniques  to  reverse  this  process.  That  is. 
use  pixel  intensities,  contours  between  the  intensity  levels,  object  models,  and  reflectance 
properties  of  light  to  recognize  an  obj-ct  in  a scene.  It  is  difficult  to  analyze  complex  pictures 
because  of  the  interaction  of  several  light  sources  and  the  reflection  from  several  small 
surfaces. 


5.  Object  Models 

Object  models  are  used  to  predict  the  positions  and  appearances  of  features  For 
example,  if  the  model  includes  descriptions  of  the  shapes  and  surfaces  of  the  object,  it  is 
possible  to  predict  the  positions,  shapes,  and  colors  of  the  corresponding  regions  in  a picture 
of  the  object 

Object  modelling  at  this  level  of  detail  is  quite  complex.  The  representation  depends 
upon  the  purpose  of  the  model.  It  makes  a difference  whether  the  object  is  going  to  be 
looked  at,  picked  up,  or  painted.  Models  may  include  one  or  all  of  the  following  facets:  a 
rigid  inter-affixment  of  features  (e.g.  sec  [Finkel  74],  [Taylor  76],  and  [Lieberman  75b]),  a 
structural  deKription  (eg.  see  [Agin  72],  [Nevatia  74],  [Baumgart  74a],  [Grossman  75a]. 
[Miyamoto  75],  and  [PADL  74]),  an  articulation  description  (eg.  see  (Nevatia  74]).  and  a 
desciiption  of  the  surfaces  (eg.  sec  [Coons  67],  [Gorden  72],  and  [Gould  72]).  These 
individual  components  arc  reasonably  well  understood,  but  there  are  no  existing  systems  that 
provide  all  of  them. 


6.  A Constraint  Resolving  Algorithm 

A constraint  resolving  algorithm  takes  as  input  an  object  model  and  a set  of  constraints 
on  the  object,  and  produces  a list  of  the  remaining  degrees  of  freedom  and  the  resulting 
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constraints  associated  with  those  degrees  of  freedom.  For  example,  if  a coffee  cup  is  known  to 
be  sitting  upright  on  the  table  and  a vision  program  locates  the  center  of  the  hole  on  top,  the 
constraint  system  produces  a locus  of  points  that  describes  the  possible  positions  for  the 
handle  of  the  coffee  cup. 

A completely  general  constraint  resolving  system  is  difficult  to  program  because  an 
object  may  have  as  many  as  six  nonlinearly  related  degrees  of  freedom  (three  displacements 
and  three  rotations)  and  the  loci  are  often  quite  complex  subregions  of  this  6-space  Ambler 
and  Popplestone  have  investigated  a limited  set  of  constraints  for  cylinders  and  V -blocks  (see 
[Ambler  73]).  Taylor  has  developed  a linear  programming  system  that  handles  a wider  range 
of  constraints  and  produces  specific  tolerance  information  (see  [Taylor  76]).  Grossman  has 
combined  a graphics  model  with  a Monte  Carlo  simulation  technique  to  produce  the 
distributions  formed  by  a set  of  input  constraints  (see  [Crossman  76]).  All  of  these  programs 
are  steps  in  the  right  direction,  but  there  is  still  a great  deal  of  work  to  be  done. 

7.  Combination  Rules  for  Probabilities 

Combination  rules  use  a priori  probabilities  and  the  value  and  position  information 
provided  by  the  operators  to  produce  estimates  for  various  probabilities.  T ypical 
probabilities  of  interest  are  the  probability  that  an  object  is  present  (eg.  a screw),  the 
probability  that  the  current  match  is  the  correct  match,  and  the  probability  that  a set  of 
matches  is  structurally  consistent. 

Probability  analyses  have  been  used  for  numerous  tasks.  Some  of  the  formulations  that 
are  the  closest  to  the  one  developed  in  this  thesis  are:  medical  diagnosis  models  (eg  see 
[Shortliffe  75],  and  [Davis  76]),  a fault  detection  model  (eg.  see  [Nilsson  75]),  and  parts 
recognition  models  (eg.  see  [Duda]  and  [Rosen  74]).  All  of  these  models  use  Bayes’  theorem 
or  a closely  related  mechanism  to  compute  the  desired  a posteriori  probabilities. 

All  of  these  systems  have  to  make  some  strong  independence  and  conditional 
independence  assumptions  in  order  to  convert  Bayes'  theorem  into  a usable  form.  Similar 
assumptions  are  made  in  this  thesis.  The  validity  of  the  results  depend  upon  the  validity  of 
these  assumptions. 

8.  Combination  Rules  for  Position  Information 

Combination  rules  for  position  information  use  position  and  orientation  estimates  from 
individual  operators  to  produce  an  overall  estimate  for  the  location  of  the  object.  The  rules 
can  also  use  the  precisions  associated  with  each  position  estimate  to  produce  an  overall 
precision  estimate  for  the  location  of  the  object.  For  example,  if  it  is  known  that  a screw 
dispenser  is  sitting  upright  on  the  table  and  if  a vision  program  locates  two  features  on  the 
dispenser,  then  the  combination  rules  should  be  able  to  produce  an  estimate  for  the  location 


of  the  dispenser  The  precision  of  this  estimate  depends  upon  the  precision  of  the  position 
infoiniation  of  the  operators  and  on  the  structural  relationship  between  the  two  features.  If 
the  line  joining  tlie  two  features  happens  to  be  perpendicular  to  the  table,  the  overall 
contribution  of  the  two  features  will  be  less  than  if  the  line  is  parallel  to  the  table,  because  the 
plane  of  the  uncertainties  is  parallel  to  the  table. 

Combination  rules  of  this  type  are  closely  related  to  constraint  resolving  procedures 
Both*  reduce  a set  of  constraints  into  a single,  overall  description  of  the  remaining 
uncei taintics,  The  difference  is  that  the  combination  rules  are  designed  to  deal  with 
unreliable  operators.  In  particular,  they  have  to  deal  with  inconsistent  pieces  of  information, 
some  of  which  arc  seriously  in  error,  and  should  be  discarded,  and  some  of  which  are  only 
slightly  inconsistent  and  should  be  included  in  the  computation. 

Some  fitting  schemes,  such  as  least -squares,  are  well-known  and  have  been  used  in  a 
wide  vaiiety  of  tasks.  Almost  from  the  very  beginning  of  computer  vision  research,  fitting 
schemes  have  been  used  to  optimize  various  criteria.  Roberts  used  a least-squares  routine  in 
one  of  the  intei mediate  steps  involved  in  deciding  which  block  prototype  best  fit  the  observed 
data  (see  [Roberts  6?]).  Perkins  used  a least-squares  fitting  technique  within  his 
coi  ner-f indcr  to  locate  the  lines  that  form  the  corner  [Perkins  73],  Cennery  used  a 
least-squares  fitting  routine  to  calibrate  a stereo  pair  of  cameras  (see  [Gennery  75]). 

Fischlcr  and  Elschlager  have  taken  a slightly  different  ajiproach  to  the  combination 
rules  problem.  They  use  a linear  programming  scheme  to  determine  the  optimum  matching 
position  for  a structure  of  features  (see  [Fischlcr  7lb]).  Their  method  can  weight  each  feature 
and  the  links  between  the  features.  Their  technique,  however,  is  more  concerned  with  the 
position  of  the  best  overall  match  than  with  the  precision  and  confidence  associated  with  that 
match. 

9.  /<  StaUgist 

A strategist  tiies  to  construct  the  most  efficient  program  to  accomplish  a task  by  ranking 
the  available  operators  and  choosing  the  most  appropriate  control  structures.  It  bases  its 
decisions  upon  the  expected  contributions  and  costs  of  the  operators  when  applied  to  the 
specific  task.  For  example,  a strategist  may  decide  to  try  to  locate  a point  on  a large  ellipse 
and  use  the  information  from  that  operation  to  locate  the  desired  hole.  Or  the  stiategy 
program  may  decide  to  locate  a point  on  the  ellipse,  follow  the  ellipse  a short  distance  in  order 
to  refine  the  system’s  estimate  of  the  location  of  the  ellipse,  and  then  try  to  locate  the  desired 
hole 


The  artificial  intelligence  community  has  been  working  on  the  strategy  problem  for  a 
long  time  The  basic  approaches  can  be  found  in  [McCarthy  58],  [Nilsson  71],  [Pikes  71], 
[Sacerdoti  75b),  and  [Taylor  76].  Feldman  and  Spioull  have  developed  one  of  the  most 
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comprehensive  systems  to  deal  with  costs,  constraints,  and  confidences  (see  [Feldman  75]  and 
[Sproiill  77)).  Their  system  is  based  upon  a decision  theoretic  model  of  strategies.  Other 
examples  of  the  use  of  strategy  programs  within  computer  vision  research  can  be  found  in 
[Yakimovsky  73a]  and  [Garvey  76]. 

10.  An  Interactive  Programming  System 

A VV  system  that  can  be  easily  programmed  requires  a human  engineered,  top-level 
system  that  interacts  with  the  user.  This  type  of  control  program  provides  an  environment  in 
which  a user  can  experiment  with  different  operators,  arrange  VV  programs,  and  test  them 
on  trial  pictures. 

Recently  considerable  interest  has  been  shown  in  teach-by-doing  programming 
techniques  for  arm  programs  (eg.  see  [Rosen  74]).  Some  of  this  enthusiasm  has  extended  into 
vision  research.  For  example,  an  interactive  parts  recognition  system  has  been  implemented 
at  SRI  [Rosen  74]  Two  other  vision  systems  that  provide  this  type  of  user  interface  are 
described  in  [Agin  76]  and  [Garvey  76], 

//  Conclusions 

The  purpose  of  this  chapter  was  to  outline  the  set  of  capabilities  required  within  a VV 
system.  Systems  already  exist  that  provide  several  of  these  capabilities,  such  as  camera 
calibration  and  object  modelling.  However,  two  of  the  most  important  capabilities  are 
missing:  combination  rules  for  probabilities  and  combination  rules  for  position  information. 
The  next  three  chapters  develop  a set  of  these  combination  rules  for  VV. 


CHAPTER  3 


FXECllTION-TIME  COMBINATION  RULES  FOR  INSPECTION 


1 he  intiodiictory  cliapters  have  stressed  the  importance  of  combining  the  results  of 
several,  possibly  unreliable  tests  Chapters  3,  -1,  and  5 derive  combination  rules  for  VV.  The 
derivations  are  fundamental,  but  the  reader  may  wish  to  look  at  some  examples  first. 
Appendices  IV  and  V contain  extended  examples  of  the  application  of  these  combination 
rules  1 he  examples  provide  a general  feeling  for  the  behavior  of  the  rules. 

1 here  are  two  types  of  combination  rules:  execution-time  rules  and  planning-time 
lilies  7 he  fanner  is  concerned  with  combining  the  itetua/  results  of  the  operators  as  they  find 
features  1 he  latter  is  concerned  with  computing  and  combining  the  expected  contributions  of 
the  operators.  In  this  chapter  we  deal  with  execution-time  rules  needed  to  accomplish 
inspection  Execution-time  rules  for  location  are  discussed  in  the  next  chapter. 

The  combination  rules  are  incrementally  developed  in  conjunction  with  a sequence  of 
examples  designed  to  incorporate  increasing  levels  of  complexity.  Each  section,  except  the  last 
one,  extends  the  rules  to  cover  one  additional  aspect  of  the  problem.  The  last  section 
discusses  some  of  the  important  assumptions  that  are  made  in  the  first  five  sections  in  order 
to  derive  the  combination  rules 


Section  I 

OPERATOR  VALUE  INFORMATION 

Consider  the  standard  inspection  task:  decide  whether  or  not  there  is  a screw  on  the 
end  of  the  screwdriver.  For  simplicity  assume  that  normalized  cross-correlation  is  the  only 
type  of  operator  known  to  the  VV  system.  Correlation  uses  patches  from  a planning  picture 
as  features  to  be  found  in  a test  picture.  Figure  3.1  1 shows  a planning  picture  with  the  screw 
on  the  end  of  the  screwdriver  and  several  sample  pictures,  some  with  the  screw  present,  some 
with  It  missing.  Figure  3.1  2 shows  several  correlation  patches  outlined  on  top  of  the  planning 


picture.  Whenever  operator  1 is  applied  to  a sample  picture  it  locates  a best  match  with  a 
certain  value,  which  is  a function  of  the  correlation  coefficient. 

If  operator  1 is  applied  to  a sample  picture  in  which  the  screw  is  missing,  no  portion  of 
the  picture  will  match  operator  1 very  well.  Operator  1 will  still  locate  a best  match,  but  the 
correlation  coefficient  will  be  lower.  Thus,  operator  1,  if  reliable,  will  (1)  match  the  correct 
piece  of  the  screw,  if  the  screw  is  there,  and  (2)  match  some  other  feature  (with  a lower 
correlation  value)  if  the  screw  is  not  there.  This  performance  difference  is  the  basis  for 
deciding  whether  the  screw  is  there  or  not. 

Figure  3.1.3  shows  the  results  of  applying  operator  1 to  ten  different  sample  pictures 
where  the  screw  is  present.  If  the  frequency  of  these  values  is  assumed  to  follow  a normal 
distribution,  the  corresponding  distribution  can  be  approximated  from  the  experimental  mean 
and  standard  deviation  of  these  values.  The  fitted  sample  distribution  is  shown  in  figure 
3.1.4.  If  operator  I is  applied  to  several  pictures  without  the  screw,  the  resulting  values  will 
form  some  other  distribution.  A table  of  ten  such  trials  and  the  corresponding  distribution 
(again  assuming  a normal  distribution)  are  shown  in  figure  3.1.5.  The  two  frequency 
functions  are  superimposed  in  figure  3.1.6.  The  assumption  of  normality  is  not  necessary  for 
the  rules  derived  in  this  chapter.  The  assumption  is  convenient  for  displaying  example 
distributions  and  it  simplifies  some  of  the  planning-time  computations,  but  it  is  not  necessary 
for  the  execution-time  formulas.  Section  3.6  will  outline  the  potential  advantages  of  normally 
distributed  values. 

If  operator  1 is  applied  to  a test  picture  to  determine  whether  the  screw  is  there,  the 
operator  will  find  a best  match  with  some  value,  say  .93.  Based  solely  upon  operator  I, 
should  the  system  say  that  the  screw  is  there  or  not?  In  probabilistic  terms,  what  is  the 
probability  that  the  screw  is  there,  given  that  operator  I has  a value  of  .93?  Denote  that 
quantity  as  follows: 

(3.1.1)  P[<screw  there)  1 <value  of  operator  1 is  .93)]. 

Let 

(3.1.2)  On  I <the  screw  Is  on  the  end  of  the  screwdriver) 

Off  i <the  screw  is  not  on  the  end  of  the  screwdriver) 
vl  • <operator  1 returns  the  value  vl) 


then  the  basic  a priori  probability  diagram  is 


r 
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and  the  inclusion  of  P[vl]  produces 


On 

Off 

• vl 

t 

Bayes’  theorem  (eg.,  see  [Hoel  71])  is  a convenient  way  of  combining  this  a priori  probability 
I information  with  the  distribution  information  to  produce  an  estimate  for  the  probability  that 

the  screw  is  On.  Bayes’  theorem  expresses  the  desired  a posteriori  probability  in  terms  of  the 
a priori  and  conditional  probabilities  as  follows: 


(3.1.3) 


P[On|vl]  * 


P[vl|0n]*P[0n] 


P[vl|0n]*P[0n]  ♦ P[vl|Off]tP[Off] 


or 


(3.1.4) 


P[On|vl]  » 


1 


P[vl|Off>P[Off) 

1 4 

P[vl |0n]*P[0n] 


These  formulas  state  the  desired  probability  in  terms  of  probabilities  that  are  often  more 
readily  computed  The  a priori  probabilities  are  based  upon  measured  statistics  or  the 
experience  of  the  assembly  engineer.  For  example,  if  the  screwdriver  correctly  acquires  a 
Krew  nine-tenths  of  the  time,  P[0n]  is  .9.  The  density  functions  shown  in  figure  3.1.6  can  be 
used  to  compute  the  conditional  probabilities,  P[vl|0fr]  and  P[vl|On].  Since  the  functions 
are  density  functions,  the  probability  of  the  operator  producing  any  one  particular  vailue  is 
zero.  But  the  probability  of  the  operator  producing  a value  within  a certain  range  is  the 
integral  of  the  function  over  that  range.  Thus  one  way  of  estimating  the  above  ratio  for  a 
specific  value  of  the  operator  is  to  consider  a small  range  about  the  value,  compute  the  two 
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probabilities  by  integi  ation,  and  form  the  ratio.  Notice,  however,  that  as  the  width  of  the 
region  decreases,  the  a()pro,ximations  for  the  ratio  approach  the  ratio  of  the  two  values  of  the 
density  functions  at  X.  7 hat  is,  the  ratio  of  the  probabilities  can  be  replaced  by  the  ratio  of 
the  densities.  This  observation  makes  it  particularly  easy  to  compute  the  appropriate  ratio  for 
any  value  of  the  operator. 

Bayes*  theorem  can  be  extended  to  combine  the  values  of  several  operators: 

1 

(3.1.5)  P[0nlvl.v2....vN]  = 

P[vl.v2. . . .vN|0ff]*P[0ff] 

1 + 

P[vl,v2, . . .vN|0n]*P[0n] 

Let  P[vl,On]  represent  the  probability  that  the  screw  is  On  and  that  operator  number  one 
produces  the  value  vl.  Since 


P[vl,0n] 

(3.1.6)  P[vl|0n]  E 

P[0n] 


and 

P[vl,v2.0n]  P[0n,v2] 

(3.1.7)  P[vl,v2|0n]  * * * P[vllOn,vZ]  * P[v2|0n], 

P[On]  P[0n,v2] 

then,  more  generally,  the  conditional  probabilities  can  be  expanded  into: 

(3.1.8)  PCvl,v2....vNl0n]  * P[vllOn,v2,v3, . . .vN]  • P[v2lOn,v3,v4, . . .vN]  * 


* P[v(N-l)10n,vN]  * P[vN|0n]. 

If  the  vj's  are  assumed  to  be  conditionally  independent,  that  is, 

(3. 1(9)  P[vJ|On,vJ+l, . . .vN]  • P[vJ|On]  (for  all  J's) 

then  these  probabilities  reduce  to 

(3.1.10)  P[vl,v2, ...vNIOn]  « P[vl|0n]  * P[v2|0n]  * ...  * P[vN10n], 
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I 

I 

I 


and  (3.1.5)  becomes 


1 

(3.1.11)  PL0n|vl,v2,...vN]  = , 

P[0ff]  N P[v1|0ff] 

1 ♦ , j-[  

P[On]  1=1  P[v1|0n] 

The  conditional  independence  assumption  stated  in  (3.1.10)  is  a major  assumption  It 
makes  it  possible  to  reduce  the  interdependences  significantly.  But  it  also  restricts  the  type  of 
operators  that  can  be  used  within  a VV  system  that  is  based  upon  formulas  such  as  (3  1. 1 1). 
If  the  value  of  an  operator  is  not  approximately  conditionally  independent  of  the  values  of 
other  operators,  it  can  not  be  used. 

Both  (3.1.4)  and  (3.1.1 1)  make  it  clear  that  the  contribution  of  an  operator  is  the  value  of 
the  ratio. 


(3.1.12) 


P[vi|Off] 


P[v1 |0n] 


The  contribution  of  an  operator  determines  its  influence  on  the  estimate  of 
P[0n|vl,v2, . . .vH].  The  inverse  of  ratio  3.1.12, 


(3.1.13) 


P[ v1 |0n] 


P[vi|0ff] 


is  known  as  the  likelihood  ratio.  The  logarithm  of  the  likelihood  ratio  is  also  important,  as 
the  chapter  on  planning-time  combination  rules  will  show.  The  larger  the  likelihood  ratio, 
the  stronger  the  evidence  that  the  screw  is  present.  This  formulation  agrees  with  one’s 
intuition  in  several  ways.  Con.«ider  figure  3.1.7  in  which  three  values  of  the  operator  have 
been  indicated;  W,  X,  and  Y.  If  the  operator  happens  to  produce  the  value  W,  the 
likelihood  ratio  is  I.O,  and  the  estimate  for  the  probability  that  the  screw  is  there  is 
unchanged.  Any  value  to  the  left  of  W implies  a likelihood  ratio  less  than  1.0,  and  thus 
decreases  the  estimate  of  the  probability  that  the  screw  is  there.  Both  X and  Y are  to  the  left 
of  W;  both  suggest  that  the  screw  is  nor  there,  but  Y does  so  more  strongly,  as  expected. 


) 

i 


Not  all  values  to  the  right  of  W will  necessarily  suggest  that  the  screw  is  there. 
Consider  figure  S.1.8.  It  emphasizes  the  difference  between  the  two  standard  deviations  so 
that  it  becomes  clear  that  there  can  be  a region  to  the  right  of  W in  which  the  likelihood 
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ratios  are  less  than  one  Only  a small  interval  (labeled  a)  contains  values  that  suggest  that  the 
screw  is  there  The  likelihood  ratios  for  every  value  in  or  are  greater  than  one  All  other 
values  for  the  operator  produce  likelihood  ratios  less  than  one.  Figure  3.19  shows  the 
likelihood  ratios  and  the  log  likelihood  ratios  associated  with  the  distributions  shown  in 
• figure  3.1.7. 

The  formulation  of  (3.1.11)  is  computationally  convenient.  For  example,  define 

P[0ff] 

(3.1.14)  t(0)  = 

P[On] 

P[vjlOff] 

and  t(J)  = » t(j-l)  (for  j > 0) 

P[vJ10n] 

then 

1 

(3.1.15)  P[0n|v:, . . .vj]  * . 

1 ♦ t(j) 
t 

This  set  of  formulas  gives  a straightforward  way  to  incorporate  the  results  of  sequentially 
applied,  conditionally  independent  operators  incrtncntally.  In  fact,  it  is  a powerful  way  to 
combine  the  value  information  of  operators  into  a probability  that  an  object  is  present. 

For  example,  consider  the  screw  checking  task.  If  P[0n]  is  .90.  then  t(0)  is  .11111. 
Assume  that  the  density  functions  shown  in  figure  3.1.10  correspond  to  operator  I.  If  it 

(returns  a value  of  810  (represented  by  the  vertical  line  in  the  figure),  then  t(  1 ) is  .0080  and 
P[0n|vl]  is  9920.  If  the  desired  confidence  is  less  than  this  amount,  the  program  can  stop 
I applying  operators  and  make  a decision:  the  screw  is  present.  If  the  desired  confidence  is 

999,  more  information  is  needed.  Consider  the  density  functions  for  operator  2 shown  in 
* figure  3.1.1 1.  If  the  second  operator  returns  a value  of  .840,  then  t(2)  is  .000476  and 

P[On|vl,v2]  is  .9995,  which  is  sufficient  to  make  a decision. 

I 

I 


I 
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Section  2 

KNOWN  ALTERNATIVES  FOR  A FEATURE 

111  the  pievioii.<i  section,  an  operator  was  applied  over  some  portion  of  a picture  under 
the  assumption  that  there  are  only  two  possible  results:  (1)  the  screw  is  present  and  the 
operator  locates  the  a|ipropriate  piece  of  the  screw  or  (2)  the  screw  is  not  present  and  the 
operator  locates  some  other  best  match.  It  was  also  assumed  that  the  operator  was  applied 
over  the  whole  region  before  returning  the  best  match.  In  effect,  these  assumptions  guarantee 
that  the  value  return’d  by  the  operator  belongs  to  one  of  the  two  density  functions.  Or.  or  Off. 
This  result  is  pleasant  if  true,  but  there  are  several  reasons  why  these  assumptions  might  be 
unwarranted; 

(1)  There  may  be  similar  features  in  the  same  local  area  that  sometimes  appear 
better  to  the  o|ierator  than  the  proper  match  If  a similar  feature  appears 
regulaily  enough  in  sample  pictures  so  that  the  system  can  determine  the 
corresponding  density  function,  the  tejiure  will  be  called  a known  alternative. 

In  that  case  the  desired  feature  is  itself  considered  to  be  one  of  the  known 
alternatives.  Every  time  a best  match  is  found,  the  VV  program  must  decide 
which  alternative  is  being  matched.  If  a similar  feature  occurs  infrequently 
and  unpredictably,  it  will  be  referred  to  as  a surprise. 

(p)  Each  a|)plication  of  the  operator  may  be  so  expensive  that  it  is  prohibitive  to 
scan  it  ovei  the  complete  area  in  order  to  choose  the  best  match  Instead,  it 
has  to  be  sequentially  applied  until  some  reasonably  good  match  is  found.  If 
there  are  a few  similar  features  in  the  local  area,  a reasonably  good  match 
may  not  be  the  best  match,  and  hence  the  value  produced  by  the  operator 
may  not  belong  to  one  of  the  two  density  functions. 

(3)  The  measurements  made  by  the  operator  may  not  immediately  single  out  the 
best  match.  A correlation  operator  is  a special  type  of  operator,  it  only 
returns  one  value,  so  it  is  easy  to  determine  which  match  is  best.  Other 
operators  may  return  values  along  several  scales.  The  best  match  is 
experimentally  defined  to  be  the  one  that  produces  values  closest  to  the 
training  values.  For  example,  an  edge  operator  may  return  both  the 
distinctness  of  the  edge  and  the  contrast  across  the  edge.  If  the  desired  line  is 
a fuzzy  line  with  a high  contrast,  it  is  not  clear  how  to  determine  the  best 
match.  A nietric  has  to  be  defined. 

(4)  The  desired  feature  may  not  be  in  the  portion  of  the  picture  scanned  by  the 
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operator.  This  problem  may  occur  if  the  program  has  incorrectly  restricted 
the  tolerance  region  for  a match,  or  if  the  feature  has  been  obscured  for  some 
reason  The  operator  still  returns  the  location  and  value  for  the  best  match  it 
can  find,  but  such  a value  does  not  belong  to  either  of  the  densities;  no 
conclusions  can  be  drawn  about  P[0n|vl,v2, . . .vN]. 

(5)  Some  global  factor  may  change  (eg.,  one  of  the  workstation’s  lights  may  be 
out)  so  that  the  feature  appears  quite  different,  even  though  it  is  in  the 
correct  area  In  this  case  the  values  of  the  operator  may  be  raoically  different 
than  planned. 

In  this  section  the  combination  rules  are  extended  to  include  known  alternatives.  Later 
sections  will  discuss  the  extensions  necessary  for  surprises,  multiple-valued  operators,  sparsely 
applied  operators,  and  global  changes. 

Consider  the  problem  of  correctly  deciding  which  of  three  possible  line  segments  an 
edge  operator  has  located.  There  are  several  sources  of  information  (orientation,  fuzziness, 
contrast,  etc  ),  but  for  the  time  being  consider  only  one  dimension  (eg.,  contrast).  Assume  that 
during  the  training  session  the  system  gathered  enough  statistics  about  the  three  lines  to 
approximate  the  three  density  functions  associated  with  their  contrast  values.  If  an  edge  is 
found  with  a certain  contrast  in  an  actual  picture,  which  line  is  the  operator  on  (assuming 
that  there  are  only  three  possibilities)  and  what  is  the  confidence  associated  with  that 
decision?  This  question  can  be  answered  by  computing  three  probabilities:  the  probability 
that  the  operator  has  located  line  1,  the  probability  that  the  operator  has  located  line  2,  and 
the  probability  that  the  operator  has  located  line  3.  Let 


(3.2.1) 

LI 

E <operator 

1 

has 

located 

a 

point 
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line 

1> 

L2 

1 <operator 

1 

has 

located 
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1 Ine 

2> 

and  L3 

i <operator 

1 

has 

located 

a 

point 

on 

line 

3>. 

Then  Bayes’  theorem  states  that 


1 

(3.2.2)  P[Ll|v]  * 

P[vhLl]*P[^Ll] 

1 ♦ 

Ptv|Ll]*P[Ll] 


Consider  the  following  diagram 
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LI 

L2 

L3 

• V 

• 

Since 

(3.2.3)  -LI  = L2  ® L3  { ® stands  for  exclusive  OR  ), 

(3.2.4)  P[v|-Ll]  = P[v|L2]*P[L21-Ll]  + P[ v| L3 ]*P[ L3 | -LI ] . 

Formula  3.2  2 reduces  to 

1 

(3.2.5)  PfLHv]  = . 

P[v|L2]*P[L2,-Ll]  P[vlL3]*P[L3,-Ln 

1 + + 

P[vlLl]*P[Ll]  P[v|Ll]*P[Ll] 

Since  L2  and  L3  form  -LI,  each  is  contained  in  -LI.  Therefore,  (3.2  5)  can  be  further  reduced 
to 

1 

(3.2.6)  P[Lllv]  = . 

P[v|L2]*P[L2]  P[v|L3]*P[L3] 

1 + 

P[vlLl]*P[Ll]  P[v|Ll]*P[L13 

When  there  are  N known  alternatives  formula  3.2.6  can  be  generalized  to 

1 

(3.2.7)  P[Lj|v]  « . (for  lijiN) 

P[vlLi]*P[Li] 



("j  PCv|Lj]*P[Lj] 

This  formula  is  convenient  because  it  states  the  desired  probability  in  terms  of  a priori 
probabilities  and  likelihood  ratios.  Given  the  value  of  an  operator  that  has  several  known 
alternatives,  the  probability  of  eaclr  alternative  is  computed,  and  the  alternative  with  the 
largest  probability  is  the  best  match. 
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This  derivation  is  also  useful  for  an  inspection  task  in  which  there  are  two  or  three 
known  alternatives  when  the  object  is  there  and  two  or  three  known  alternatives  when  the 
object  IS  not  there  In  this  case  the  program  is  less  concerned  about  which  alternative  is  the 
best  match  than  about  the  overall  probability  that  the  object  is  there.  A derivation  similar  to 
the  one  used  above  produces  the  formula  needed  in  this  situation.  Let  fl,  f2,  ...  fM  be 
the  known  alternatives  that  might  be  matched  when  the  object  is  there  and  let  ol,  g?,  . . . 
gN  be  the  alternatives  that  are  possible  when  the  object  is  not  there.  Bayes’  theorem  states. 

1 

(3.2.8)  P[On|v]  = . 

P[v|0ff]*P[0ff] 

1 + 

P[v|0n]*P[0n] 

By  assumption 

(3.2.9)  P[On]  = P[fl]  + P[f2]  ♦ ...  + P[fM] 
and 

(3.2.10)  P[0ff]  = P[gl]  + P[g2]  ♦ ...  + P[gN]. 


Formula  3.2.8  can  be  expanded  into 


(3.2.11) 

P[On|v]  = 


1 


P[v|ol3*P[gI  ] + P[v|g2]*P[g2]  ♦ ...  ♦ P[v|gM]*P[gN] 

1 ♦ 

P[v|fl]»P[fl]  + P[v|f2]*P[f2]  ♦ ...  + P[v|fM]«P[fM] 


or 


1 

(3.2.12)  P[0n|v]  . , 

Y.  P[v|gi]*Ptgi] 

ISi^N 

1 ♦ 

^ P[v|fi]»P[f1] 

This  formula  gathers  all  of  the  evidence  for  and  against  the  proposition  On  and  forms 
a ratio  between  them.  To  use  this  formula  requires  a great  deal  of  knowledge  about  what  can 


Page  “18 


3.2 


\ 


be  c\pecte(i  in  a luntnnr  picliiie  In  particular,  this  knowledge  includes  the  set  of  possible 
alternatives,  theii  values,  and  their  a ptior!  probabilities.  Within  the  context  of 
programmable  assembly  this  infoi matiou  is  often  available  because  the  environment  is  highly 
constrained  and  a piogram  has  the  opportunity  to  watch  several  examples  of  the  assembly. 


Consider  the  application  of  formula  3.2.12  to  the  screw  checking  example  When  the 
screw  IS  not  on  the  end,  one  of  the  operators  (say  operator  1 that  tries  to  locate  the  tip  of  the 
screw)  may  retuin  a diffeient  set  of  values  depending  upon  whether  or  not  the  detent  at  the 
end  of  the  sciewdiivct  is  showing.  (The  detent  is  the  spring-loaded  ballbearing  that  holds 
the  sciew  on  the  end  ) It  appears  in  the  picture  only  if  the  screwdriver  is  oriented  in  such  a 
way  that  aims  the  detent  at  the  camera.  Thus,  there  are  two  known  alternatives  for  operator  1 
when  the  sciew  is  not  on  the  end:  (1)  the  detent  is  showing  and  (2)  the  detent  is  not  showing. 
When  the  sciew  is  piesent,  there  is  still  only  one  known  alternative.  Thus,  the  a prion 
probabilities  for  operator  1 are: 


(3.2.13)  P[On]  = P[5crew] 

P[Off]  = P[Dctent]  + P[No  Detent], 


which  can  be  diagianied  as  follows. 


On 

Off 

Detent 

Screw 

No 

Detent 

This  diagram  is  just  for  operator  1.  The  other  operators  may  not  be  affected  by  the  detent. 

If  operatoi  1 is  the  first  operator  to  be  applied,  formula  3.2. 12  simplifies  to: 

(3.2. H) 

1 

f 

P[Onlvl]  = — — — 

P[vl  lDetent]iiP[Oetent]  ♦ P[vl|No  Detent]*P[No  Detent] 
1 + 


P[ vl lScrew]*P[Screw] 
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S.2 

Assume  'hat 

(3.?. 15)  P[Scrcw]  s .90 

P[Delent]  = .03 
P[No  Detent]  = .07 

and  that  the  density  functions  shown  in  figure  3 2.1  are  the  functions  for  o|ierator  1.  If 
operator  1 returns  a value  of  .81,  then  P[0n|vl]  is  .9910.  Figure  3.22  shows  the  three  known 
alternatives  for  operator  1 and  three  typical  values  produced  by  operator  I Each  column  of 
the  table  contains  PfTiplv],  P[No  Detentiv],  and  P[Detent|v].  Given  a value,  vl.  for 
oper.'.tcr  1.  the  known  alternative  with  the  largest  value  of  P[<known  alternativc>  | vl] 
IS  tiie  best  match  for  that  value.  For  e.xample,  given  the  value  .47,  the  best  match  is  the 
Detent.  1 hat  is,  based  upon  the  a prion  probabilities  and  the  training  information,  a value 
of  .47  for  operator  1 implies  that  the  most  likely  feature  being  matched  is  the  detent  in  the 
end  of  the  screwdriver.  The  second  best  choice  is  the  end  of  the  screwdriver  without  the 
detent. 
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I igure  ?.2  2 

I oiiitiil.i  2 12  c.iii  l>e  e.isily  cxtciidfci  to  incoi  jiorate  the  results  of  several  operators,  all 
of  wIikIi  iniy  li  ive  l.imv.’ii  aliem.oivcs  Assiiine  that  thcie  are  K operators.  Let  fj,l; 
fj.?;  ...  hr  the  fJj  f.iu.nni  alternatives  for  the  jth  opetator  when  the  object  is 

pi. sent  I't  ci.i,l;  .-.ttij.Hj  be  the  MJ  known  alternatives  for  the  jth  operator 

whrn  thr  objeet  is  nut  jne-rnt  7 lien 


(3.2.10) 

f’l  On  1 vl , v2  , . . . vK  ] = 


1 


(K-1)  ^ P[vjlgj, i ]*P[gj,  i ] 

PfOn]  K ISisNj 

* f 1 


(K-1)  j=l  ^ P[vj|fj,i]*P[fj,1] 
P[0ff]  IstisNj 


7 he  ex|ionfnt  (k-l)  ajipears  because  the  e)i,|)i ession  lor  each  of  the  K operators  produces  a 
factor  of 


(3.2.17) 


P[0n] 


PL  Off] 


and  the  ratio  of  i?  /nieii  prob.ibiliiies  in  formula  3. 2. is  cancels  one  of  them. 
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Section  3 
SURPRISES 

1 he  main  assumption  of  the  last  section  is  that  all  of  the  alternatives  are  known  and 
chaiactenzed  m advance.  Sometimes,  however,  operators  match  unknown  features  and  return 
unusual  values  Such  unknown  and  unexplained  matches  will  be  referred  to  as  surprises. 
The  values  produced  by  surprises  can  not  be  accounted  for  by  the  usual  density  functions 

1 here  are  several  possible  causes  for  an  unusual  match  (some  global  change,  the  feature 
IS  not  present,  or  a sur|jrise),  so  the  values  produced  for  such  a match  should  not  be  used  to 
alter  the  overall  confidence  estimate.  The  values  may  contribute  to  other  considerations  (such 
as  a global  etror  decision),  but  they  should  not  be  blindly  cranked  through  the  combination 
rules 


There  are  two  ways  to  deal  with  unusual  values:  (1)  filter  out  particularly  bad  values 
and  (2)  scale  down  the  potential  contribution  (in  the  probability  computations)  of  any 
operator  that  is  known  to  find  surprises  1 he  first  method  involves  a check  on  each  value 
produced  by  an  opeiator  to  make  sure  that  it  is  reasonable  for  at  least  one  of  the  known 
alternatives  For  example,  if  a value  is  not  within  three  standard  deviations  of  the  mean  of 
at  least  one  known  alternative,  classify  it  as  an  unusual  value.  This  method  is  generally  only 
applicable  when  some  unexpected  event  occurs,  such  as  one  of  the  lights  burns  out  at  the 
workstation  so  that  the  pictures  of  the  scene  are  significantly  different  than  expected. 

T he  second  method  is  to  lower  the  possible  contribution  of  unreliable  operators.  This 
method  is  based  upon  a simple  rule:  an  operator  that  finds  surprises  should  not  be  trusted  as 
much  as  one  that  doesn’t.  1 he  assumption  that  all  of  the  alternatives  are  known  has  been 
expressed  in  foimula  3.2  9 

(3.3.1)  P[On]  = P[fl]  + P[f2]  ♦ ...  P[fN]. 

If  the  operator  occasionally  locates  surpiises,  a better  model  is 

(3.3.?)  P[On]  = P[fl]  + P[f2]  ♦ ...  P[fN]  ♦ P[s] 

where  Pts)  is  the  a priori  probability  of  finding  a surprise.  To  reflect  this  model  in  the 
piobability  computations  roniiies  a density  function  to  be  associated  with  the  surprises 
What  should  the  foim  of  this  density  be?  If  surprises  can  randomly  produce  any  value  for 
the  o|)erator.  one  reasonable  assumjition  is  that  the  surprise  density  has  a rectangular 
distribution.  If  the  filtering  method  (i.e.,  method  1 mentioned  above)  is  applied,  the  range  of 
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the  ret  cli'.li  ihiitum  c.iii  t«-  re'inctril  lo  tlic  mictv.il  between  the  smallest  reasonable 

value  foi  the  (ipeiatoi  and  the  latpest  tfi'nnable  value  heure  3 ? I a shows  three  density 
functions,  one  loi  the  case  in  which  the  scicw  is  (ircsent  and  two  known  altet natives  when  the 
sciew'  IS  not  (nesent  If  the  0(>etaior  occasionally  locates  surprises,  a rectangular  density 
function  IS  added,  as  shown  in  figine  3 3 1b 

1 he  denviiy  function  foi  suipiises  can  be  incorjiorated  into  the  confidence  computation 
in  a stiaightforwaid  way  Since  a surpiise  may  occur  whether  or  not  the  object  is  present,  the 
new  possibility  is  included  in  both  the  numerator  and  the  denominator  However,  the 
piobability  of  a sui prise  may  be  diffeient  when  the  screw  is  present  than  when  it  is  not 
That  IS.  P[SurprisclOn]  may  be  diffeient  from  P[SurpriselOff ] In  order  to  provide  for 
this  possibility,  diffeient  quantities  are  included  in  the  numerator  and  denominator.  If  s 
lepiesents  a surprise,  foinuila  3.2  12  can  be  lestated  as; 


(3.3.3) 


P[On|v]  = 


1 


N 

P[v|s,0ff]*P[s,0ff]  ♦ y P[v|gi ]*P[gi  ] 
i = l 


1 a- 


M 

P[v|s,0n]*P[s,0n]  ♦ V P[vl f i ]*P[ f i ] 
i = l 


1 he  additional  density  function  restricts  the  contribution  of  the  suspect  operator.  The 
ojrerator  can  not  be  as  strongly  for  or  against  the  proposition  On  as  it  could  be  when  all  of  the 
altern.atives  were  known  For  example,  consider  operator  I mentioned  in  the  last  section  The 
a fitiori  probabilities  are 

(3.3.4)  P[On]  = P[On, Surprise]  ♦ P[Screw] 

P[Off]  = P[ Off, Surprise]  + P[0etent]  ♦ P[No  Detent] 

and  the  corresponding  diagram  is 


P.ti;e  !'-t 


? 3 


Cm 

Off 

r 

On,  Surpr  i s-f 

Off. Surpr i se 

Detent 

Screw 

No 

Detent 

Fotmul.i  3.3.3  leiliices  to 
(3.3.i.)  t’lOnlvX]  = 

1 


1 


i 


I 


('[  vX|  Of  f , Surp]*(’[Of  f ,5iirp]  + f'[  vX)  Del  ]*!'[  Del  ] + F'[vX|No  Dct]*F'[No  Dot] 

1 + 

f’[  vXIOn  ,Suri>]*r[On,Surp]  * P[  vXI  Scrcw]*P[  Screw] 

Assume  tfiat 


(3.3.0)  r’[  Screw]  = .88 

P[Del]  = .02 
8[Ho  Oct]  = .00 
P[0rf,5urp]  = .02. 
and  P[0n,Surp]  = .02. 

Since  P[  Off  ..Siirp  1,  P(On,Siirp],  P[vl  [On.Surp],  and  P[vl  jOff.Surp]  are  constants, 
foimiila  3.3.fi  i '.duces  to 


(3.3.7) 

P[On|vl]  = 


1 


.0318  + PI vl i0ct]*Ptnct]  + P[vllNo  Oct]*P[No  Det] 


1 ♦ 


.0318  + P[vl |Scrcw]*P[ Screw] 


I lie  maximum  value  of  PlOn|vl]  is  achieved  when  operator  1 returns  a value  that 
miiiiiTiires  both  P[vl|Dct]and  P[vl|Ko0ct].  At  th.at  point  P[0n | vl  ] is  .7754  This  value 
IS  significantly  less  than  the  maxiniuin  value  of  I 0.  which  is  possible  if  the  surprises  are  not 
incorporated  into  the  formula.  So  incorporating;  surprises  into  the  formulas  for  operators 
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known  to  produce  surprising  values  reduces  the  operators'  contributions  toward  the  overall 
probabilities,  thereby  implementing  method  2 above. 

If  the  density  functions  shown  in  figure  3.3.l.b  are  the  functions  for  operator  I,  and 
ofjerator  I returns  a. value  of  .63,  the  conditional  probabilities  associated  with  the  alternatives 
are: 

(3.3.8)  P[Tip| .63]  = .0011 

P[Det| .63]  = .1338 

P[No  Detl .63]  = .7096 
and  P[Surprisc| .63]  = .1556. 

The  best  alternative  is  No  Detent  and  the  overall  probability  that  the  screw  is  on  is  .0789. 

The  incorporation  of  surprises  also  means  that  sometimes  the  best  match  may  be  a 
surprise.  For  example,  if  operator  1 happens  to  return  a value  of  .40. 
P[Surprisc|  .40]=. 8869,  which  means  that  the  best  match  is  the  surprise. 

Formula  3 can  be  extended  to  combine  the  results  of  several  operators,  each  of 
which  may  have  known  alternatives  and/or  sui prises.  Let  fj,0  be  the  surprise  associated 
with  the  jth  operator  when  the  object  is  there  and  let  gj,0  be  the  surprise  for  the  jth 
operator  when  the  object  is  not  there.  Then  the  formula  can  be  written  as: 

(3.3.9) 

1 

P[0n|vl,v2. . . .vK]  = 

X!  P[vj|gj, 1 ]*P[gj. i ] 

P[0ff]  K P[0n]  OilSNj 

1 ♦ * []  ( » ) 

P[0n]  j=l  P[0ff]  Y.  P[vjlfJ.n*P[fj.1] 

OSiSMj 

1 his  extension  to  include  surprises  means  that  there  are  three  possible  outcomes 
whenever  an  operator  is  applied;  (1)  the  value  is  outside  the  reasonable  range,  (2)  the  value  is 
reasonable,  but  the  best  match  is  a surprise,  or  (3)  the  value  is  reasonable  and  the  best  match 
is  a known  alternative.  The  higher-level  interpretation,  if  any,  of  the  unusual  values  and 
surprises  will  be  discussed  in  a later  chapter. 
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Section  4 

MULTI PLF.-VAHJED  OPERATORS 


Sonic  opcistors  icttini  more  tlian  one  vaUic;  tlic  description  of  what  they  have  fcuind 
contains  vaincs  alonr;  several  scales.  For  example,  a textiue  operator  may  desoibe  a local 
iceion  in  terms  of  its  si/e.  density,  and  periodicity,  it  has  already  been  mentioned  that  edge 
opeiatois  often  letiiin  two  or  three  values.  When  ch.aling  with  such  opeiators  one  wants  to 
combine  all  of  the  available  infoiniation  into  one  piobability  that  the  object  is  present,  oi  to 
determine  the  best  alternative.  Again  liayesian  probability  piovides  a way  to  make  iliis 
combinaticm  Consider  an  ins|icction  task  and  one  0|jerator  that  returns  M values,  xl,  x2, 

. . . and  xM  T hen  the  standard  Ljayesian  formula  is 


(3.4.1) 


P[ On  I X 1 , x2 , . . .xM] 


1 


P[xl,x2, . . .xM  I Off]  P[Off ] 

1 + * 

P[xl,x2, . . .xM  1 On]  P[On] 


If  the  values  hap|jen  to  be  conditionally  independent  of  each  other,  the  usual  reduction  yields 


(3.4.2) 


P[On|xl ,x2, . . .xM]  = 


1 


P[0I  f] 

1 + * 

P[0n] 


M 


n 
£ 1 


P[xi lOff ] 


P[xilOn] 


T hese  foiniul.as  can  be  extended  to  include  several  operators,  each  of  which  may  return 
seveial  values  Assume  that  there  aie  N operators  and  each  operatoi  returns  Mj  values  (M.)  St 
1)  Let  xj,l:  xj,2;  ...  xj.Mj  be  the  Mj  values  returned  by  the  jth  operator.  If  the 
values  foi  one  operator  are  interdependent,  but  the  values  of  separate  operators  are 
conditionally  independent,  then 
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(3.4.3) 

PlOn  I (xl.l;  xl,2:  ...  xl.Ml) (xN.l;  xN.2;  ...  xN.MN)  ] = 

1 


P[0ff]  N P[  (xj,l:xj.2:...;xj.Mj)  | Off] 

1 + * 

P[On]  j=l  P[  (xj,l;xj,2:...:xj,Mj)  | On] 

If  all  of  the  values  are  conditionally  independent  of  each  other  this  formula  collapses  back  to 
the  previous  formula  (with  a suitable  renunibciing  of  the  x’s). 

Foimula  3 4 3 can  be  further  extended  to  include  operators  that  have  several  known 
alternatives  and  even  surprises.  Assume  that  the  values  for  one  operator  are  interdependent, 
but  that  the  values  of  separate  operators  arc  conditionally  independent.  Let  there  be  K 
operators.  Let  the  jth  operator  have  Mj  known  alternatives  when  the  object  is  there,  and  N j 
known  alternatives  when  the  object  is  not  there.  Let  MO  and  NO  represent  the  surprises 
Assume  that  the  jth  operator  returns  Rj  values  as  a description  of  what  it  finds.  Then  the 
appropriate  formula  is: 

(3.4.4) 

P[On  I (xl,l;xl,2;...;xl,Rl) (xN. 1 :xN,2 ; . . . jxN.RN) ] = 

1 


^ P[(x1,l:x1,2:...;x1,Ri)|gJ.1]*P[gj,1] 
P[Off]  K P[On]  OSl^Nj 

1 ♦ • fl  ( » ) 

PtOn]  j»i  P[Off]  ^ P[(xi,l:xi.2:...:xi.Ri)|fj,i]*P[fj.i] 

OSfSMJ 

To  use  operators  that  return  several  interdependent  values  requires  enough  information 
to  approximate  the  multi-dimensional  density  functions.  Once  this  has  been  done,  the  ratio 
of  density  values  can  be  used  in  place  of  the  ratio  of  probabilities,  just  as  in  the 
one-dimensional  case. 

Since  the  expression  (xj,!;  xj,2:  ...  xj.Rj)  can  be  validly  substituted  for  vj  in 
any  of  the  derivations  that  follow,  the  remaining  derivations  will  only  be  concerned  with 
single-valued  operators.  The  formulas  apply  to  multiple-valued  operators,  but  for  notational 
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sim()licity  they  will  not  bo  stated  in  their  full  generality. 


Section  5 

I’OSn  ION  INFOUMATION 

1 he  loral  value  irifoi mation  produced  by  an  operator  is  important,  but  the  relative 
stiuctiirc  of  the  niatches  i.s  crucial  in  verification  vir-iun.  This  section  describes  two  methods 
that  iiiccn  poi  ate  stmctiiial  infoi  mation  into  the  execution-time  forinulas  for  inspection.  One 
method  uses  a set  oi  matches  to  cieteiminc  the  most  consistent  location  for  the  object  and  then 
decides  lu.w  hf.ely  that  location  is  in  light  of  the  initial  task  constraints  and  the  training 
information  "I  he  semnd  method  simjily  determines  the  number  of  reasonably  consistent 
matches,  given  the  fact  that  the  features  are  part  of  a rigid,  thrcc-dimensiorial  object  A 
major  diawback  of  the  first  method  is  that  the  amount  of  computation  goes  up  exponentially 
as  the  numhei  of  operatuis  increases  The  second  method  is  a simple  heuristic  to  avoid  this 
large  amount  of  computation,  but  it  docs  not  take  full  advantage  of  the  available  information 

ITguie  ? 'i  1 a shows  the  jiosiiions  of  four  typical  features  in  a planning  picture 
Assume  thai  the  task  is  to  cleteimine  the  change  from  the  planning  picture  to  the  test  picture 
and  th,»t  the  change  is  mainly  an  X-Y  shift  If  the  four  operators  are  ajjplied  to  a test 
pictuie  and  they  locate  their  best  matches  at  the  positions  shown  in  figure  3.6.1.b,  the  relative 
stiuctuic  c>t  the  matches  appeals  to  be  coiiect.  A Ica.st-squares  fitting  routine  (or  some  other 
fitting  loutiiie)  can  be  used  to  [Uoduce  an  estimate  for  the  shift  and  an  estimate  for  the 
goodness  ol  fit  In  this  example,  the  residual  errors  are  quite  small  (as  shown  in  figure 
3 'i  l.c).  Nince  this  is  a guod  fit,  one  would  say  that  the  matches  are  structurally  consistent. 
Howevei.  if  the  foui  matches  aic  found  at  the  positions  shown  in  figure  3.5. Id.  the  best  fit 
would  contain  laige  criurs  (see  figuie  3.5  l.e).  Iii  this  case  one  would  probably  be  suspicious 
of  at  least  one  of  the  niatebes 

"I  lie  im|.lication  is  that  the  residual  errors  arc  a function  of  the  structural  consistency  of 
the  set  of  matches  1 he  less  consistent  the  matches,  the  larger  the  eirors.  The  sum  of  the 
squares  ot  the  eiicjis  is  commcnily  used  to  mcasute  this  type  of  consistency.  It  is  a convenient 
measure  b'C.uise  ihcie  aie  well-known  techniques  for  minimizing  it  It  is  also  appealing 
i because  the  disti ibuticm  of  the  sum  of  the  squares  of  the  errors  is  known  to  be  a Chi-square 

disti  ibutioii  It  the  eiiois  are  normally  distributed  [Craybill  61].  Since  measurement  errors  arc 
known  to  be  noimally  distributed  for  a large  number  of  situations,  the  use  of  least -squares 
techniques  looks  quite  |)iomi5ing 

1 he  thecuem  that  specifies  the  distributictn  of  the  sum  of  the  squares  of  the  errors  can 
be  stated  as  follows 
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Figure  3.5. 1.a 


Figure  3.5. l.b 
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Figure  3.5. 1.c 


Figure  3.5.  !.d 


Figure  3.5. 1.e 


Figure  3.5. 1.f 
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THEOREM:  If  there  arc  N linear  equations  relating  the  actual 

n.itchiny  positions  with  the  planned  positions  and  if  there  are  R 
parfiiiicters  to  be  adjusted  in  the  transformation  from  the  planned 
to  actual  positions,  the  sum  of  the  squares  of  the  errors  (for 
normally  distributed  errors)  forms  a Chi-square  distribution 
with  (N-R)  defirees  of  freedom. 

One  ini|ilicaiiun  of  tins  thrcueni  is  that  a C.hi-sc|u:ne  test  esn  be  applied  to  a particular  sum 
of  squaies  to  cleieiniinc  whether  it  reirrcsents  a consistent  transformation  between  the  jjlanncd 
and  actual  positions  If  the  test  indicates  that  the  set  of  matches  is  not  consistent,  it  is  possible 
to  determine  winch  maicli  is  the  least  consistent.  For  example,  in  figure  3.5. he,  the  hexagon  is 
the  least  consisitiit  feaiiiie.  The  least  consistent  match  can  be  temporarily  left  out  of  the 
solution  and  nnothcr  least-squares  fit  can  be  conijuitcd  If  the  new  fit  is  significantly  better 
than  the  jjievions  one,  the  least  consistent  match  can  be  peimancntly  removed  from  the  set  of 
matches  This  culling  of  bad  niatclies  can  continue  until  a consistent  set  of  matches  has  been 
found  1 bus,  another  measure  of  the  consistency  of  a set  of  matches  is  the  percentage  of 
matches  deemed  consistent  by  this  culling  piocedurc.  Figure  3.5  I f shows  the  best  fit  after  the 
least  consistent  match  has  been  discarded.  The  leinsining  thiec  marches  are  structuia)ly 
consistent 

As  expected,  the  concept  of  stiiictural  consistency  is  an  important  aspect  of  verification 
But  how  should  it  be  integiated  with  the  value  information?  It  is  possible  to  extend  the 
conditional  piohalnlity  foiniulas  to  include  the  |jositions  of  the  matches  in  addition  to  the 
values  of  the  matches.  Let 

(3.5.1)  pi  c <thc  position  of  operator  i's  match), 

then  Bayes’  theoiem  states; 


1 

(3.5.2)  P[ On  I vl , . . . vN, pi , . . .pN]  = . 

Ptvl,...vlJ,p]...pN|Off]  P[0ff] 

1 + * 

P[vl, . . .vN,pl . . .pN|0n]  P[0n] 

If  the  Vi’s  are  assumed  to  be  conditionally  independent  of  the  pi’s  (and  each  other),  formula 
3 5.2  reduces  to 
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(3.5.3) 

1 

P[On| vl , . . .vN,pl , . . .pN]  = . 

f'[pl . . .pNIOff  ] P[0ff]  N P[vi|Off] 

1 + * * P]  

P[pl . . .pNIOn]  P[On]  i=l  P[vi|On] 

The  assumption  that  the  vi’s  arc  conditionally  independent  of  the  pi’s  means  that  the  value 
of  an  operator  is  independent  of  the  location  of  the  match.  That  is,  if  an  operator  locates  the 
same  feature  at  different  positions  in  different  test  pictures,  it  is  expected  to  produce  the  same 
value  In  programmable  assembly  this  is  generally  a reasonable  assumption  except  when 
different  positions  consistently  produce  different  lighting  conditions.  For  example,  if  a 
shadow  happens  to  fall  on  a feature  when  the  object  is  oriented  in  a certain  way,  the  value  of 
the  operator  that  tries  to  find  that  feature  will  depend  upon  the  orientation  of  the  object 

The  assumption  one  does  not  want  to  make  is  that  the  pi’s  are  conditionally 
independent  of  each  other.  Such  an  assumption  would  completely  ignore  structural 
consistency,  which  is  precisely  what  the  mathematics  is  intended  to  capture.  But  what  is  the 
value  of 


P[pl,p2, . . .pN  I Off] 

(3.5.4)  ■? 

P[pl ,p2, . . .pN  I On] 

First,  consider  a simpler  version  of  (3.5.4)  that  is  only  concerned  with  the  positions  of  two 
operators 


(3.5.5) 


P[pl,p2  I Off] 


P[pl.p2  I On] 


The  probability  P[pl,p2|0ff]  is  the  probability  that  operator  1 will  find  its  match  at  the 
position  pi  and  that  operator  2 will  find  its  match  at  the  position  p2  Ratio  3.5.5  can  be 
rewritten  as 


(3.5.6) 


P[pl.p2.0ff]*P[0n] 


P[pl,p2,0n]*P[0ff] 


If  P[Off  ] could  be  expressed  simply  as: 


(3.5.7) 


P[Off]  * P[X]  ♦ P[Y]  ♦ P[Z], 
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then  P[  p 1 , Of  f ] ciiuld  be  cx|iiesscd 

(3.5.8)  ('[pi, Off]  = l’[pl.X]  + P[pl.Y]  + P[d1.Z] 
and  P[  pi , p2  , Of  f J could  be  expressed  as 

(3.5.9)  P[pl.p2.0ff]  = Plpl.p2.X]  + P[pl.p2,Y]  + P[pl,p2.Z]. 

Howcvei,  P(Off  1 IS  mote  coni|ilicatcd  than  that.  It  is  a sum  of  several  conjunctions.  Assume 
tliat  the  fust  operator  has  ml  known  aliern.itives  when  the  screw  is  Off  and  nl  known 
alternatives  wlirn  tlie  screw  is  On.  Let  gl,l;  gl,2;  ...  gl,ml  be  the  ml  known 
alternatives  when  tlie  screw  is  Off  and  let  fl,l;  fl,2:  ...  fl,nl  be  the  nl  known 
alternatives  when  the  screw  is  On.  Similarly  let  g2 , 1 ; g2 , 2 ; ...  g2,m2  and  f2,l;  f2,2; 
. . . f2.n2  be  tire  known  alternatives  for  the  second  operator.  Then  P[0ff]  and  P[0n]  can 
be  expressed  as  follows 

nl  m2 

(3.5.10)  P[Off]  - y ^ Plvl.Ji  92, k], 

j=l  k=l 

Given  this  expression  tor  the  piobability  P[0ff].  the  analogous  formula  to  (3.6.9)  is 

ml  m2 

(3.5.11)  P[pl,p2.0ff]  = V ^ P[pl ;p2:gl. j:g2.k]. 

j=l  k=l 

Then  ratio  3.56  can  be  rewritten  as 

ml  m2 

y y P[pl;  p2;  gl.j;  g2.k] 

P[0n]  j=l  k=l 

(3.5.12)  * . 

P[0ff]  nl  n2 

V V P[pl;  p2;  fl,j;  f2,k]. 
j=l  k=l 

Thus,  ratio  3.5.’l  has  bf<ri  reduced  to  the  evaluation  of  several  expressions  of  the  form 
P[  pj  :pk ; f 1 , j ; f 2 , k ].  which  represents  the  probability  that  operator  1 has  found  known 
alternative  fl.J  at  position  pj  and  operator  2 has  found  known  alternative  f2,k  at  position 


pk 


In  Older  to  see  how  estimates  for  probabilities  such  as  P[  pi  ;o2  ; fl , j ; f2 , k ] might  be 
com|)iitcd.  consider  figure  3 ?>  ?,  which  is  a picture  of  a screw  taken  during  the  standard  screw 
checking  task.  In  that  task  the  location  of  the  tip  of  the  screw  is  assumed  to  be  known  within 
plus  01  minus  one  quarter  inch  along  each  axis  and  the  orientation  of  the  screw  is  assumed  to 
be  within  plus  or  minus  fifteen  degrees  of  vertical.  Figure  3 fi.2  shows  two  features  on  the 
sciew.  If  two  opeiators  tiying  to  locate  these  features  happen  to  find  their  best  matches  at  the 
locations  shown  in  figure  3 5 3.  what  is  the  probability  that  they  have  located  the  correct 
features?  The  chance  is  probably  small  because  it  would  require  the  screw  to  be  rotated  HO 
degrees  from  vertical,  which  is  highly  unlikely  since  the  initial  uncertainty  is  only  plus  or 
minus  fifteen  degrees.  If  the  two  matching  positions  are  at  the  locations  shown  in  figure  3. 5.4, 
the  probability  that  the  matches  are  correct  is  significantly  higher  since  the  impliec. 
displacements  and  rotation  are  within  the  expected  ranges  The  important  idea  is  that  the 
estimate  of  the  probability  P[pl  :p2;fl,  j:f2,k]  can  be  based  upon  the  likelihood  that  the 
object  would  be  at  the  location  implied  by  the  set  of  matches. 

The  likelihood  that  an  object  is  at  a specific  location  can  be  expressed  in  terms  of  the 
parameters  that  describe  the  location.  For  example,  consider  a task  in  which  the  uncertainties 
are  pseudo-planar  (i  e.,  the  unknown  parameters  are  X and  Y displacements,  dx  and  dy,  a 
rotation  change,  dt.  and  a scale  change,  ds,).  Statistics  can  be  gathered  at  training  time  that 
describe  the  expected  density  functions  for  these  parameters  (see  figure  3.5.5).  If  two  matches 
imply  that  dx=  . 1 , dy=-,2,  dt  = -2.3,  and  ds=l  .02,  the  probability  P[  pi  ;p2  ; f 1 , j ; f? , k ] 
can  be  expressed  as 

(3.5.13)  P[pl;  p2:  flj;  f2,k]  = P[  dx=  . 1 ;dy=- .2  ;dt  = -2 .3  :ds=  1 .02  ] . 

To  a first  approximation  the  parameters  can  be  treated  independently.  Thus. 

(3.5.14)  P[pl;p2:fl,j;f2,k]  = P[dx= . 1 ]*P[dy=- .2 ]*P[  dt=-2 .3]*P[ ds=  1 .02,  ] . 

The  value  of  each  of  the  component  probabilities  can  be  determined  by  integrating  the 
appropriate  density  function  over  a small  interval  centered  about  the  nominal  value. 

This  step  completes  the  demonstration  that  the  position  information  can  be 
incorporated  into  the  conditional  probability  formulas  and  that  there  are  reasonable 
techniques  for  estimating  the  necessary  probabilities.  The  final  formula  for  two  operators  is; 
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(3.5.15)  (•IOn|vl,v2,pl,p2]  = 


] 


ml  m2 

V V p[pl;  p2;  gl,j;  g2 , g ] 

N P[vi|OffJ  j=l  k=l 

1 4 PI * 

i=l  PlvilOn]  nl  n2 

V P[pl;  p2:  fl.j;  f?.k]. 
j=l  k=l 

Tlie  value  terms  arc  based  upon  the  operators'  evaluations  of  the  appearances  of  the  features 
and  the  position  leims  are  based  upon  the  likelihood  of  the  object  being  at  the  location 
implied  by  the  position  of  the  matches. 

luimula  3 5 15  can  be  evpauded  to  include  N operators,  but  the  combiriatoi  lal 
ex[jlosion  limits  its  usefulness,  lor  e,\3n)pk,  if  five  operators  each  have  three  known 
alternatives  when  the  sciew  is  picsciit  and  iluce  alternatives  when  the  screw  is  missing,  -IfiS 
se|iarate  piob.ibilities  would  have  to  be  computed.  Ten  operators  would  require  1ISI18 
terms  1 his  tact  suggests  that  formula  3.5.15  should  only  be  used  to  evaluate  the  structural 
consistency  of  small  subsets  of  tlie  total  number  of  operators.  Section  4.2  will  discuss  the  use 
of  this  type  of  formula  to  assign  known  alternatives  to  the  results  of  two  or  three  operators 
In  effect,  the  foiniula  makes  it  possible  to  choose  the  most  consistent  pattern  of  alternatives 

for  a small  number  of  operators 

« ' 

One  way  to  avoid  the  computation  costs  inherent  in  the  straightforward  application  of 
formula  3.5.15  to  a large  number  of  operators  is  to  use  some  other  technique  to  approximate 
the  ratio 


P[pl,p2  I Off] 


(3.5.16) 


P[pl,p2  I On] 

One  heuristic  that  has  piovcd  to  be  experimentally  useful  is  to  replace  (3.5.16)  by 
<perccnlaoc  of  consistent  features,  oiven  Off> 


(3.5.17) 


<percenta{)c  of  consistent  features,  given  0n> 


This  ratio  is  only  a crude  approximation  to  the  ratio  of  the  probabilities,  but  it  is  useful 
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because  it  contributes  a factor  based  upon  the  structural  consistency  of  the  matches 

Since  the  total  number  of  possible  matches  is  the  same  for  the  two  cases,  On  and  Off, 
the  ratio  of  percentages  reduces  to 

<nuinhcr  of  consistent  features,  given  Off> 

(3.5.18)  . 

<numbcr  of  consistent  features,  given  0n> 

1 hus,  the  contribution  of  structural  consistency  in  the  probability  formulas  can  be 
transformed  into  a ratio  of  the  numbers  of  consistent  matches. 

In  order  to  compute  a value  for  the  ratio  in  (3.5.18)  the  least-squares  culling  routine  is 
a|)plicd  twice:  once  assuming  that  the  screw  is  present  and  once  assuming  that  the  screw  is 
missing  The  input  for  each  application  is  a list  of  pairs: 

(3.5.19)  <p1anned  feature  position,  matching  position  of  the  opGrator>. 

^ The  planned  feature  positions  for  the  operators  depend  upon  the  assumption  about  the  screw. 

If  the  sciew  is  assumed  to  be  piesent,  one  set  of  planning  positions  is  used.  If  the  screw  is 
assumed  to  be  missing,  a different  set  of  |jlanning  positions  is  used,  because  the  operators  are 
expected  to  find  different  features  when  the  screw  is  missing. 

The  set  of  features  for  each  assumption  forms  a geometric  pattern  (or  structure).  The 
least-squares  culling  routine  is  used  to  measure  the  agieement  between  a planning  pattern 
and  an  actual  pattern  of  matches.  The  ratio  in  (3.5.18)  measures  the  relative  agreement.  7'his 
relative  measure  is  the  contiibution  of  the  position  information  toward  the  overall  confidence 

In  most  cases  the  stiucture  of  the  planning  features  is  significantly  different  when  the 
screw  IS  On  than  when  the  screw  is  Off.  This  difference  essentially  guarantees  that  the  ratio 
in  (3  5 18)  will  not  be  close  to  1.0.  As  the  number  of  matches  included  in  the  fit  increases,  the 
structures  for  the  two  situations  become  more  distinct.  This  behavior  is  important  if  the 
position  information  is  to  distinguish  between  the  two  situations. 

All  important  assumption  of  this  discussion  so  far  has  been  that  the  operators  match 
unique  features,  one  for  On  and  one  for  Off  If  there  are  several  known  alternatives  for  an 
operator,  the  piogiam  has  to  decide  which  is  the  best  match  for  that  operator  and  assign  that 
match  to  the  ojierator  so  the  least-squares  culling  routine  can  be  called.  The  basic  formul.a 
used  to  determine  the  best  alternative  was  developed  in  section  3.2.  It  is 
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(3.5.?0)  (‘[Ljlv]  = (for  all  j). 

Pi:v|Li]*P[Ll] 

1 + V 

P[v|Lj J*P[Lj] 

11  thcie  h.T[i|ioii  to  l.ir  two  oi  oiriic  slici n.ttivcs  with  ap|jioximately  equal  probabilities  of 
beiiie,  the  best  nt.itcli,  the  least -squares  cullin>;  pioccciuic  can  be  extended  as  follows: 
whenever  the  first  choice  is  af>out  to  be  discarded  (because  it  is  the  least  consistent  matcli), 
icplace  it  with  niiotiici  one  of  the  a|)pioxiniatcly  equal  choices.  This  extension  increases  the 
complexity  of  the  least-squares  culling  loutinc,  but  it  piovides  an  automatic  way  of  giving  an 
ojaeratoi  a second  cliance  whenever  there  is  more  than  one  possible  explanation  for  its  results. 

1 he  incoi poratiem  of  the  position  information  does  not  alter  the  ease  with  which  the 
pi obabilities  can  be  coniputc.’cl.  Seipicntially  acquiicd  information  can  still  be  included  very 
nicely  Since  the  least-squares  culling  |jroccriurc  can  not  be  applied  until  some  minimum 
numbei  of  l^.^^nles  has  bt'i:ii  located,  the  position  information  can  not  contribute  anything 
until  that  minimum  number  has  been  reached.  T he  minimum  number  dcjrends  upon  the 
number  of  /laiameieis  being  adjusted,  the  number  of  equations  contributed  by  each 
opei'atoi/featuie  pair,  and  the  number  of  independence  conditions.  For  example,  if  the 
least-squat  es  method  is  pet  forming  a planar  fit,  there  are  three  parameters.  dX,  dY,  and  da. 
Since  each  con  elation  feature  and  each  point-on-a-line  featui  e contributes  two  equations,  any 
two  of  these  features  would  be  sufficient  Three  or  four  would  be  better  because  the 
least-squares  technique  performs  better  v.'hen  the  parameters  arc  over-constrained.  Since  this 
IS  true,  it  may  be  advantageous  to  increase  the  number  of  matches  above  the  theoretical 
minimum  before  using  this  method  to  incorporate  position  information. 

If  there  aif  several  known  alternatives  foi  an  operator,  the  program  may  make  a 
mistake  when  it  assigns  an  alternative  to  the  results  of  an  operator.  That  is,  given  the  value 
and  position  infoimaiion  of  a match  for  the  operator,  the  piogram  may  decide  that  the 
operator  located  alternative  fl.l.  when  in  fact,  it  located  alternative  fl,2.  Since  the  program 
does  not  know  whether  it  is  correct  or  not,  one  possibility  is  to  use  the  probability  associated 
with  the  best  match  as  the  ex|)ccted  number  of  good  matches  contributed  by  the  operator. 
Thus,  It  the  prohahility  associated  with  the  best  alternative  is  .8,  the  operator  contributes  .8  of 
a good  matcli  towaid  the  desired  minimum. 

ITgure  ?.5.f>  outlines  the  general  method  suggested  by  this  section.  One  operator  after 
another  is  applied  until  the  accumulated  value  infoimation  indicates  that  sufficient  features 
have  been  located,  then  the  least-squaics  method  is  applied.  Additional  features  arc  added 
until  the  confidence  leaches  the  desired  limit  1 his  algorithm  could  form  the  basis  for  an 
inspection  system  it  can  be  used  to  decide  if  a gasket  is  already  on,  to  decide  if  a hole  has 
been  dulled,  oi  to  decide  if  the  expected  subasK-mbly  has  been  added 


Sfction  0 
ASSLIMl'TIONS 


'1  liis.  scciicin  cli^cusM's  the  «s!.iim|itiotis  ilist  aic  rd-jiiiiccl  by  the  combination  rules  used 
to  com|iute  the  |iuib:tljiliiics  of  iiiteiost.  1 lie.-^e  assumptions  are  fundamental  assum()iioiis 
about  the  class  o)  tasl^s  rrferied  to  as  verification  vision  tasks  and  about  the  probabilistic 
methods  used  to  moilel  such  tasks.  An  exam|jlc  of  such  an  assumption  is  the  conditional 
mclependence  of  the  value  and  position  information  of  the  operators.  If  this  assumption  is 
nor  approMmaiely  true  for  a paiticular  task,  none  of  the  Dayesian  probability  formulas 
develcipcd  in  this  chapter  can  be  a|iplicd,  tlieii  pi i coiidiiions  are  not  satisfied. 

I he  assumptions  have  been  classified  into  three  types:  (1)  Bayesian  probability 
assumptions,  (2)  value  distribution  assumptions,  and  (3)  conditional  independence 
assumptions  f.ach  type  will  be  discussed  in  a separate  subsection. 

I.  Dayestan  Fiolmlnlities 

r.aycs'  theoicm  .states  a desired  a postoUvi  probability  in  terms  of  the  a priori  and 
conditional  piobabilities 


(3. 6. 1.1) 


P[H|v]  = 


P[v|H]*P[H] 


l>[v|H]*P[ll]  + P[v|-,H]*P[-.H] 


This  formula  is  convenient  because  the  conditional  probabilities  P[v|H]  and  P[v|-iM]  are 
^.eneially  easici  to  measure  (or  estimate)  than  PLH|v].  However,  Shortliffe  and  Buchanan 
(see  [Slioriliile  7')]  and  (Nilsson  7fi])  have  pointed  out  two  related  problems  involved  in 
applying,  Bayes'  theoiem  to  vaiious  decision  tasks  1 he  first  is  that  one  must  be  careful  to 
specify  the  set  c>f  possible  events  and  their  relationship  to  the  propositions  of  intetest,  11  and 
-iH  In  paiticulai,  Shoitliffe,  Buchanan,  and  NiKsoii  weie  conceined  about  the  case  in  which 
H IS  a compound  pioposition  and  the  desired  piobability  is  P[vl-.H].  What  is  the  meaning  of 
7 lie  second  pioblem  is  that  the  amount  of  st  nistics  required  to  estimate  the  desired 
probabilities  may  be  piohib|tivc,  even  if  it  is  clear  which  statistics  should  be  gathered. 

In  piogi arnmable  assembly  the  fiist  pioblem  is  generally  easy  to  solve  because  the 
enviionment  is  highly  controlled  and  pieclictable.  l or  example,  consider  the  task  of  deciding 
whether  oi  not  a caiburetor  has  been  attached  to  an  engine  assembly.  Assume  that  there  are 
three  possible  events: 


(3.6.1  .2) 


A E <cnrliiirct.or  of  typo  X is  attached), 

8 E <Ciirt/urctor  of  typo  Y is  attached), 
and  C E <no  carburetor  is  attached). 

Consider  .in  opciatoi  that  bctiavcs  as  iollows;  when  a carburetor  of  type  X is  attached,  it 
locates  one  of  two  features,  (1  or  f2,  when  a carburetor  of  type  Y is  attached,  it  locates  feature 
f3;  and  when  theie  is  no  caiburetor  attaclicd,  it  locates  one  of  two  features,  f4  or  f&.  The 
coi  responding,  a fiiiori  probabilities  can  be  diagramed  as  follows; 


H 4H 


A B C 


fl 

f3 

fA 

f2 

f5 

P[  V 1 H j and  P[  V 1 -iH  ] can  be  expanded  into 

(3. 6. 1.3)  PfvIHj  = P[v|fl]*P[fl3  a-  P[v|f2]*P[f2]  ♦ P[v|f3]*P[f3] 

and  P[v|-,H]  = PIv|f4]*P[fn3  + P[  v|  f 53*P[  f53 . 


The  same  enumeration  procedure  can  be  applied  when  H is  a compound  proposition. 
For  example,  assume  that  the  engine  casing,  mentioned  in  the  previous  example,  may  or  may 
not  be  painted  and  that  the  a priori  piobabilities  of  the  possible  events  are: 


A B C 


f2 

fA 

PAINTED 

fl 

f3 

f5 

UNPAINTED 

f ; 

f3 

f7 

fC. 

It  tllC  |lHi|lCt^ltlOII  of  llllCICM  IS 


(3.6. 1 .4)  H s A n <PAINTE0>, 

P[  V I H ] .-^ncl  l’[  V I -.11  ] Crfi)  be  cxiJicsseci  as  tollows. 

(3. 6. 1.5)  ('[v|H]  = Plvir^3*P[fP]  + P[vlfl.PAINTED]*P[fl, PAINTED] 

and  P[v|-,tl]  = PLvlf3J*P[f3]  + P[  v 1 14  ]*P[  r4  ] + P[  v 1 f 5 ]*P[  fS  ] + 
P[v|fo]*P[f6]  + P[v|f7]*P[f7]  + 

Pt  v|  n ,uNPAiN7(:nj*p[  n .unpainteoj. 

1 Inis.  Ill-'  til  SI  [jiublriii  call  be  solved  by  carefully  eninrierating,  the  |jossible  events  and  then 
iiitet  .icticins 

Tven  thoiie.li  it  is  e.isy  to  solve  the  fust  piobleni  within  ptogiammable  assembly,  it  is 
clifticiili  to  avoid  the  second.  Whrnev  t theie  .ne  several  known  alternatives,  the  tiainiiig 
session  h.as  to  g.iihcr  si.itistics  for  all  of  t!ie  alici natives  and  all  of  the  interactions  between  the 
alternatives.  1 he  amuiiiit  of  statistic.!,  information  iei|uiied  and  the  number  of  tiials  needed 
to  piodiice  valid  estimates  for  the  statistics  de|jcnd  upon  the  number  of  alternatives  and  the 
c,\|iccted  disti  ibiitions  for  the  inforination.  '1  hese  quantities  will  be  discussed  in  the  chapter 
describing  the  the  ptogramming  and  training'  phases. 

2 I'niiif  Distiibutions 

'I  hroiigboiit  the  development  oi  tlie  formulas  a normal  distribution  has  been  assumed 
foi  the  value  information  of  the  operators.  That  is,  the  values  associated  with  an  alternative 
weie  as.siimed  to  have  a iioimal  distribution.  T his  assumption,  however,  is  not  necessary  to 
compute  the  likelihood  ratios 


(3.6.P.  1 ) 


P[vi  lOri] 


P[ VI lOff ] 


Any  disti  ibution  is  sutficicnt.  It  is  even  possible  to  use  the  histogram  of  values  produced  at 
tiaining  time  as  the  distribution,  as  lung  as  there  is  a sufficient  number  of  trials. 

A iioinial  disti  ibution  was  assumed  in  the  derivations  because  it  is  a good  model  foi 
some  of  the  opeiators  and  it  is  convenient  for  displaying  example  distributions.  However,  if 
tfie  values  of  an  opeiator  ate  not  riorrnally  disttibuted,  iheie  may  be  a change  of  variable  that 
can  conveit  them  into  an  approximately  normal  distribution.  A later  portion  of  this  section 
will  discuss  a change  of  variable  that  converts  cot  relation  values  into  a distribution  that  is 


Page  73 


a|jpio\imatcly  iioi  mal 

If  values  of  ati  opciatoi  are  known  to  follow  some  distribution  other  than  a normal 
distribution,  it  is  easy  to  incorporate  the  new  distribution  into  the  execution-time  formulas 
1 he  only  information  needed  iii  addition  to  the  density  function  is  a specification  for  the 
interval  of  riasonahtr  values  What  values  of  the  operator  should  be  classified  as  unusual 
and  hence  should  be  filtered  out  (see  section  3.3)?  For  normal  distributions  it  is  easy  to 
specify  an  interval  in  teims  of  the  number  of  standard  deviations  away  from  the  mean 
Other  distributions  require  some  other  specification  for  the  interval  of  reasonable  values. 

Any  distiibution  can  be  used  for  the  valtie  itiformation  of  an  operator.  For  exam/de,  if 
some  a priori  information  implies  that  tiie  distribution  for  a particular  operator  is  a gamma 
distribution,  a gamma  distiibuiicn  can  be  substituted  into  the  appropriate  formulas.  If  the 
training  results  imply  that  the  distribution  is  tiot  one  of  the  standard  distributions,  the  density 
function  defined  by  the  histogram  can  be  used  in  the  formulas. 

Some  of  these  other  distributions  have  pioperties  that  make  them  good  models  for  the 
o|)cratois  used  in  VV.  Pot  example,  the  gamma  distribution  is  asymmetrical;  it  can  model  a 
wider  range  of  distributions  than  a normal  distiibution,  and  yet  its  foim  c;in  be  easily 
determined  from  the  experimental  mean  and  standard  deviation  of  the  training  samples 

A Change  of  I'aiiahlf  Jor  Cotrdation  I'alua 

One  opeiator  that  is  known  to  produce  a non-normal  distribution  is  cross-correlation 
(see  [Hoel  71]).  Consider  the  following  formula  for  the  correlation  coefficient: 

N 

y (Xi-Mx)*(Yi-My) 

1 = 1 

(3. 6.?. 2)  r = , 

N * Sx  * Sy 

where  Xi  and  Yi  are  jointly  normally  distributed,  Mx  and  My  are  the  sample  means  of  X and 
Y,  respectively,  and  Sx  and  Sy  ate  the  sample  standard  deviations.  It  would  be  possible  to  use 
the  actual  distribution  of  r,  but  there  is  a convenient  change  of  variable  that  converts  r into 
a distribution  that  is  approximately  normal  The  change  of  variable  is 


1 1 X r 


r 


Pat;e  7^ 


3 6.2 


1 he  mean  of  the  new  ciistiibution  is 

1 1 + a 

(3.6.?.'f)  Mz  = * log^ 

2 1 - a 

wheie  Cl  icjnesrnis  the  throictical  value  of  the  coucl.ition  coefficient.  The  standard  deviation 
loi  tl»e  new  dicti  iLuiiion  is 

1 

(3.0.2.b)  S/  = , 

sqrl(N  - 3) 

wfiere  N is  the  luinihei  of  samples  used  to  compute  r. 

The  coiiclation  ojiciator  im|»lenirurcd  by  Hans  P Moravec  at  Stanford  and  used  in 
this  w’ciik  behaves  aciotding  to  this  theoiy  [Moiavec  76].  Consider  figure  3.62.1.  Figure 
36.2  la  IS  a hi'togiam  o:  titty  coirclation  coeificicnt  values.  T he  values  are  the  results  of 
ap/)lyin>  the  same  coiiilation  ojiciaiui  to  fifty  dilfeient  jjictures  of  a scene  for  one,VV  task. 
I he  interval  sire  along  the  horiroiual  asis  of  the  graph  is  one-half  of  the  sample  standard 
deviation  As  piedicted,  the  coiiclation  vnlues  form  a skewed  distribution  (with  a theoretical 
uppei  limit  of  I 0)  The  chi-siyuaie  value  is  based  ujion  the  eleven  intervals  centered  about 
the  s«miple  mean.  Figure  3G.2  l.b  is  the  histogiam  of  values  produced  by  the  change  of 
variable  in  foimula  3.6  2.3.  Figure  3 6 2.1c  is  the  histogram  that  would  be  expected  if  the 
samjile  foimecl  a peifeci  normal  distribution. 

'1  he  chi-si]uare  value  chops  significantly  fiom  2b. •!  (with  eight  degrees  of  fieedom)  to 
f 3 (with  eight  degiees  ot  tieedom)  lor  the  new  distribution.  The  improvement  is  not  always 
that  diamatic,  but  the  change  of  v.iiial>le  often  tiansforms  a skewed  distribution  into  a 
distribution  closci  to  noimal.  Considci  iiguirc  3.6.22,  a scatter  diagram  of  the  pairs: 

(3. 6. 2.0)  (<chi-S()unrc  of  raw  vnluc.s),  <chi-squarc  of  changed  values>). 

Any  point  to  the  light  of  the  diagonal  line  repiesents  a case  in  which  the  change  of  variable 
made  the  disti ibutioii  for  the  values  of  an  operator  look  more  like  a normal  distribution 
(accoiding  to  the  chi-square  test).  The  change  of  variable  only  slightly  degrades  the 
chi-square  value  in  the  few  cases  that  it  makes  the  distribution  worse  A point  in  the  shaded 
area  of  figure  3 6 22  represents  an  operator  whose  distribution  was  improved  significa.ntly 
Before  the  change  of  variable  the  chi-square  test  (at  the  5*/.  level)  rejected  the  hypothesis  that 
the  sample  could  have  come  from  a noirnal  dish ibiition.  After  the  change  of  variable  the 
chi-sqiiaie  test  indicated  that  it  was  plausible  for  the  sample  to  have  come  from  a normal 
distribution. 
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Figure  3.6.2  l.a 
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Figure  3.6.2  l.c 


36.2 


Page  77 


The  question  about  which  distribution  to  use  to  model  the  results  of  an  o(ierator  is  a 
hard  one  The  chi-square  test  used  above  is  helpful,  but  it  is  mainly  a method  to  reject 
proposed  models. 


7.  Conditional  Independence 


1 he  derivations  of  several  of  the  formulas  depend  upon  two  important  assumptions 
about  the  conditional  independence  of  the  values  of  the  operators:  (1)  the  value  of  an 
0|)ei  ator  is  conditionally  independent  of  the  values  of  the  other  operators  and  (2)  the  value  of 
an  ojicrator  is  conditionally  independent  of  the  position  of  its  match.  Both  of  these 
assuni|)rions  are  instrumental  in  simplifying  the  relevant  formulas.  For  example  they  make  it 
possible  to  simplify  formula  3 5 2: 


(3.6.3. 1 ) P[On| vl , . . .vN.pl . . . .pN] 


1 


P[vl, ...vN,pl...pN|Off]  p[orr] 

1 + * 

P[ vl , . . . vN, pi . . .pN |0n ] P[On] 


into 

(3. 6. 3. 2) 

1 

P[On I vl , . . . vN.pl , . . .pN ] = 

P[pl . . .pN|Off ] PtOff]  N P[vilorf] 

1 + * * []  

P[pl...pN|0n]  P[On]  i=l  P[vi|On] 

The  assumptions  significantly  reduce  the  number  of  dependencies  within  the  conditional 
probabilities  and  make  them  feasible  to  com|)Utc.  The  formulas  are  designed  to  be  as  flexible 
as  possible  and  still  be  computable  If  the  assumptions  are  not  true  for  an  operator,  formula 
3.6  3.2  can  not  be  used  to  estimate  the  desired  jiiobabilities 

Roth  of  the  assumptions  depend  upon  the  values  of  the  operators  and  the  values  of  the 
operators  are  based  upon  the  appearances  of  the  features.  Unfortunately,  there  are  several 
reasons  wliy  the  appearance  of  a feature  might  change  from  one  picture  to  the  next: 

(1)  1 he  feature  itself  may  be  different  For  example,  in  assembly 
tasks  all  pump  bases  are  not  exactly  the  same,  so  all  screw  holes 
are  not  the  exactly  same  In  photo-interpretation  (abbreviated 
F’l)  features,  such  as  roads,  may  be  changed 


t 

I 
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(2)  Tlic  iHisitioii  and  oiirntation  of  of,(jccts  in  the  picture  may 
cftange  In  assembly  the  pump  bases  may  not  seat  exactly  the 
same  way  in  the  vise 

(3)  "I  be  ligliting  may  be  clilfeimt  In  t’l  tlie  sun  may  be  in  a 
difleicnt  location,  causinp,  different  shadows  and  glares.  In 
piogt ammable  asseiubly  lij^.lits  can  bo  controlled  more  easily,  but 
they  still  may  vary  slightly.  , 

(•1)  ■]  he  position  and  orientation  of  the  camera  may  be  different. 

In  a'.'.rmbly  tlie  camera  may  be  used  for  mote  than  one  task  in 
such  a way  that  it  must  bo  repositioned  before  each  task, 
kepcisitionirig  is  tiot  c.xact.  In  PI  the  position  of  the  plane  or 
satellite  may  be  different 

(3)  1 be  sensitivity  of  the  camera  may  be  different.  Cameras  have 

inteinal  jjarameters  such  as  target  voltage  that  change  over 
time 

(C)  1 he  c.imcra  noise  level  is  variable 

In  effect,  the  two  conditional  independence  as.sumptions  state  that  none  of  these 
variables  change  the  expected  clisii ibuiion  of  values  produced  by  an  operator  in  VV.  Some 
Ojjerators,  of  course,  do  depend  upon  one  or  meue  of  these  variables.  This  fact  raises  an 
important  question. 

Given  a specific  VV  task,  is  there  a way  to  determine 
whether  or  not  the  assumptions  hold  for  a particular 
combination  of  operators? 

The  remainder  of  this  section  develops  some  insight  into  this  question, 

1 he  first  assumption  states  that  the  value  of  an  operator  is  conditionally  independent 
of  the  values  of  tlie  other  operators,  e g.. 


(3. 6. 3. 3) 


PLv2|0n,vl]  = PLvZlOn], 


36  3 
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I 

A 


This  formula  says  that  given  On,  the  probability  of  operator  2 producing  the  value  v2  is  the 
same  whether  or  not  the  value  of  operator  1 is  known  Formula  3.6  3.3  is  equivalent  to 

P[v2,0n,vl] 

(3. 6. 3. 4)  = P[v2|0n], 

P[ On , vl  ] 

P[  vl , v2  lOn  ] 

(3. 6. 3. 5)  = P[v2|0n]. 

P[vl jOn] 


and 

(3. 6. 3. 6)  P[vl,v210n]  = P[vl|On]  * P[v210n]. 

1 huf.  it  IS  possible  to  check  the  first  assumption  for  two  operators  by  (1)  gathering  the 
statistics  necessary  to  form  the  distributions  associated  with  the  three  conditional  probalulities 
in  (3  6.3  6)  and  (2)  appling  a test,  such  as  the  Chi-square  test,  to  decide  if  the  distribution 
formed  by  the  convolution  of  the  distributions  for  P[vl|0n]  and  P[v2|0n]  is  the  same  as  the 
distribution  for  P[vl  ,v2|0n]  The  second  assumption  can  be  checked  in  a similar  manner. 

1 he  first  assumirtion  is  often  true  in  \'V  because  different  operators  depend  upon 
different  properties  of  the  picture  The  piobability  of  producing  a certain  value  for  a 
correlation  ojieiator  is  often  unaffected  by  the  results  of  a previously  applied  edge  operator. 
An  obvious  case  in  which  the  assumption  is  not  true  is  when  operator  1 and  operator  2 are 
both  correlation  operators  and  they  oveilap.  Knowing  the  value  of  one  operator  certainly 
alteis  the  possible  values  for  the  second.  Similarly,  an  edge  and  correlation  test  for  the  same 
corner  will  be  inter-dependent.  Fortunaicly,  since  most  of  the  objects  in  programmable 
assemble  have  several  inteicsting  features,  ovcrlaj)ping  operators  can  be  avoided 

The  second  assumption  states  that  the  appearance  of  a feature  on  an  object  does  not 
change  as  the  object  moves  aioiinci  within  its  tolerance  volume.  Put  another  way,  if  an 
operator  is  applied  to  several  different  pictures,  and  it  locates  the  same  known  alternative  in 
each,  the  value  returned  by  the  ojicrator  is  independent  of  the  location  of  the  alternative  in 
the  picture  This  is  usually  true  in  VV  because,  by  assumption,  the  changes  are  so  small  that 
the  appearance  of  a feature  is  essentially  constant 

However,  there  are  two  situations  in  which  the  second  assumption  might  be  false.  The 
first  is  when  a small  change  in  one  of  the  variables  in  the  transform  causes  a shadow  to  fall 
on  a feature  At  some  locations  the  feature  is  in  a shadow  and  at  others  it  is  not  The  value 
of  almost  any  operator  attempting  to  locate  such  a feature  would  depend  upon  whether  the 
feature  is  in  a shadow  or  not  Hence  the  value  of  the  operator  would  depend  upon  the 
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position  of  tlie  feauiie  and  the  assumption  would  be  false.  The  second  situation  arises  when 
a small  change  m po'-ition  causes  a diamatic  change  in  tlie  appearance  of  a feature  For 
examjjle,  one  pic'uic  may  show  a sciew  hole  that  is  pailially  occluded  on  the  left  by  a shaft 
and  a second  pictuie  ol  the  same  hole  may  show  the  shaft  occluding  the  hole  on  the  right 

r.oth  o!  these  situations  lead  to  operators  that  produce  bivariate  (or  at  least  high 
vai  lance)  density  functions.  One  peak  is  produced  by  the  pictures  showing  the  feature  in  the 
shadciw  and  the  othei  peak  is  piodticcd  by  the  pictures  showing  it  in  the  light.  Since  the 
exi'ccted  conliibution  for  such  operatois  is  generally  low,  the  automatic  ranking  scheme  will 
place  this  type  of  opeiator  neai  the  botnam  of  the  let  of  |jotential  o|jerators  to  be  used  in  a 
task  It  the  \' V system  is  interactive,  a programmer  can  also  discard  any  features  of  this  type 
suggested  by  the  system  If  an  o|)erator  is  particuhii  ly  imijortant  it  is  possible  to  break  up  the 
task  into  twti  or  thiee  siihtasks,  each  of  which  satisfies  the  assumption.  Taylor  has  used  this 
technique  vriihin  his  constraint  re.-olving  sysicm  whenever  the  angular  uncertainties  are 
greater  than  a few  degrees  [Taylor  76}. 
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CHAPTER  4 

I XFCHTION-I  IME  COMIHNATION  RULES  FOR  LOCATION 


If  the  veiificalion  vision  system  is  tryinp;  to  locate,  not  inspect,  an  object,  there  are  two 
impoitant  quantities-  (I)  an  estimate  for  the  location  of  the  object  and  (2)  the  |jrecisioii 
associ.ated  with  that  estimate.  In  the  context  of  VV  the  location  of  an  object  refers  to  the 
position  and  orientation  of  the  object's  coordinate  system  in  terms  of  some  other  coordinate 
system  (eg.,  the  workstation  coordinate  system).  Usually  there  is  some  point  or  feature  on  tfie 
object  of  paiticulai  interest,  eg.,  the  center  of  a hole  or  the  tip  of  a screw.  Such  a point  will 
be  referred  to  as  a point  of  intocst. 

1 he  previous  chaptei  briefly  mentioned  that  a least-squares  technique  can  be  used  to 
combine  a set  of  |danned  positions  with  a set  of  cot  responding  measured  positions  to  |)ioduce 
an  estimate  for  the  ti ansfoi mation  between  tliem  Given  this  transformation  and  the  planned 
location  of  the  object,  it  is  easy  to  compute  the  current  estimate  for  the  location  of  the  object. 
The  least-squares  technique  can  also  jiroduce  the  standard  deviations  associated  with  the 
estimates  for  the  individual  parameters  in  the  transform.  These  standard  deviations  can  be 
combined  to  pioduce  an  estimate  tor  the  juecision. 

1 here  aie  other  metiics  and  fitting  |irocedurcs  beside  least-squares  techniques.  The 
least-squaies  appioach  was  chosen  for  the  current  implementation  of  the  VV  system  because 
It  provides  the  desiied  location  and  piecision  information  and  it  is  a well-known  technique 

The  application  of  the  least-squares  routine  depends  upon  the  correspondence  between 
the  matching  points  and  the  pi  inning  features  If  the  correspondence  is  correct,  the  estimate 
for  the  location  of  the  object  and  the  associated  precision  will  be  correct,  II  it  is  not  correct,  it 
is  possible  to  determine  a (seemingly)  stnictinally  consistent  subset  of  the  features,  which 
leads  to  an  incoiicct  estimate  for  the  location  of  the  object.  This  problem  only  arises  when 
there  are  several  known  alternatives  for  the  features  or  when  the  operators  find  suijjiises  To 
avoid  making  this  mistake  an  ovciall  estimate  of  the  probability  that  the  object  is  within  the 
stated  piecision  is  needed.  Such  an  estimate  can  be  based  upon  the  value  and  position 
information  of  the  operators 

1 fiis  chapter  begins  with  a detailed  exjilanation  of  the  least-squares  method  and  its 
application  to  tfie  VV  jiioblem  of  producing  location  and  precision  estimates  The  emjjhasis 
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on  till’  th.  oiiIkmI  a'-|iOCi5.  of  tlic  lr,i<,!-5i.]u.iits  iri>-iliod.  The  piactical  a|i|)licaiions  of  tliis 
mrthoti  air  ih  lauii^ii ati'ii  m V,  whith  conr.iins  an  annotated  trace  of  a piogramnicr 

Intel  actiiif,  v.iili  thr  cm  lent  im|ili  niriitation  of  the  VV  system.  The  programmer  sets  up  a 
N \'  piOi;i,im  to  loc.air  a sciew  hole  and  tesis  it  on  sevcial  e.\amples.  The  trace  demorisiiaies 
the  cuinpm.ttion  ol  prccisicui,  tlie  ciilhnf,  of  bad  matches,  and  the  reduction  of  tolerance 
legions 

'I  he  second  s>  i.iiun  of  this  chapter  clescribcs  an  example  in  which  the  results  of  the 
least-squares  mtihocl  arc  iiiconect  and  tlicn  discusses  two  methods  to  estimate  the  confidence 
associated  with  a siatoment  of  inecision. 


Section  1 

1)1  -n  IIMIN'INC.  I’ltl  ClSlON 

'1  Ins  sriiion  |iiisrnts  a gmcral  method  lor  |iei  forming  nonlinear  generalized 
le.iSt-sqnai*'.  a l|usinniits  A maiv  portion  of  this  discussion  is  a restatement  of  an  internal 
pajici  at  tin-  St.mfoid  Aiiilici.al  Inti  llii.eiice  I’lOject  v.iitteii  by  Donald  F.  Geiinery  entitled 
"Least-Squaies  >St.:i eo- C.amci  a Calibiation"  [Ginnery  7S],  The  method  uses  partial 
deiivaiivcs  to  appiu.Mmatc  the  jjiohlern  uncici  the  general  linear  hypothesis  model  ol 
statistics,  and  then  iteiatcs  to  achieve  the  c.sact  solution.  For  more  detailed  information  sec 
[Crayhill  CD] 

'I  hr  iioiaiional  conventions  aie  the  following.  Capital  letters  denote  matrices.  Vectors 
aie  repi  esenicil  by  column  matrices.  A parucular  clement  of  a matrix  is  represented  by  the 
ceil  I espciiidiiig  lowei-casr  letin  followed  by  the  appiopriate  indices.  T he  transpose  of  a 
matri.x:  A is  ch  noted  by  A’,  and  the  invcise  of  A is  denoted  by  A~.  Multiplication  (either 
sc<iiii  Ol  mat/ix)  is  drnoifd  by  an  asterisk. 

I.rt  the  vi'fior  G cUnote  a set  of  m unknown  parameters  for  which  values  are  desired 
Let  the  vecioi  U be  a set  of  n scalar  ipi.aiititirs  (nem)  that  arc  functions  of  G and  can  be 
measuied  u'lih  some  euoi.  let  I-  icpiesrnt  the  vector  of  n functions  that  relate  elements  in  G 
with  G Given  an  csiimatr  lor  G,  1(G)  produces  an  estimate  loi  U.  Finally  let  the  vector  V 
iipiesint  the  n ii'-idnals  (l  e.  the  nnexpl.mied  eiiois)  that  remain  between  U and  an  estimate 
produetd  by  F(G)  T bus 

(4.1.1)  U = F(G)  + V. 


T he  goal  is  to  minin.ize  a function  ol  V by  modifying  G 
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In  verification  vision  G is  the  set  of  (>aiameters  in  the  transform  that  maps  the  planned 
|.iositions  of  the  featiiics  into  their  matching  positiotu  (i.e.,  the  planned  positions  into  the 
measiiird  positions).  Ty|iical  elements  in  C are  the  displacement  in  X,  dx,  the  displacement 
in  dr.  and  the  iinl^nown  rotation  about  the  Z-axis,  da.  Different  o|jeratoi7featiirc  jrairs 
contiibute  dillnciit  components  to  U and  f For  example,  when  the  transform  is  planar  (so 
that  the  unknown  |)aiameteis  are  dx.cly.  and  da),  a correlation  operator/feature  pair 
contiilnites  two  measured  values  to  U:  the  \ and  Y components  of  the  match  (let  them  be 
referred  to  as  Xn  and  Ym)  The  corresponding  functions  in  F are: 

(4.1.Z)  Xc  = (Xp-Xc)*COS{c]a)  - ( Yp-Yc)*SIN(da)  ♦ dx  Xc 

Ye  = ( Xp-Xc)*SIN(da)  + ( Yp-Yc)*COS(da)  ♦ dy  ♦ Yc 

where  (Xc.Yc)  is  the  center  of  rotation  for  da,  (Xp,Yp)  is  the  planned  position  for  the 
coil  elation  patch,  and  (Xc.Yc)  is  the  transformed  position  of  (Xp,Yp).  The  transformed 
position  of  (Xp,Yp)  is  the  estimate  for  (Xp,Yp)'s  position  in  the  current  picture.  The  two 
residuals  that  would  be  associated  with  a correlation  feature  are 

(4.1.3)  Xn  - Xe 

and  Ym  - Ye. 

1 hese  residuals  are  the  components  of  V The  goal,  of  course,  is  to  use  the  measured  values 
to  impiove  the  estimates  for  the  jjarameiers. 

7 he  quadratic  form 

(4.1.4)  q = V'*W*V 

IS  the  criterion  of  optimization  that  is  to  be  minimized.  W denotes  an  n by  n weight  matrix. 
If  W is  the  inverse  of  the  covaiiance  matrix  of  the  errors  in  the  observations,  the  result  will 
be  the  maximum  likelihood  (in  the  F space)  solution  if  the  errors  have  a normal  distribution, 
it  VV  is  a diagonal  matrix,  which  indicates  no  con  elation  between  errors  in  the  different 
observations,  the  quadratic  foim  reduces  to  a weighted  sum  of  the  squares  of  the  elements  of 
V Thus  the  pioblem  as  stated  here  can  be  said  to  be  a generalized  least-squares  adjustment 

1 he  difficulty  in  obtaining  a solution  to  the  above  problem  lies  in  the  fact  that  F in 
(4  1 I)  is  a nonlinear  function,  and  thus  in  general  there  is  no  closed-form  solution  One  way 
of  solving  the  problem  is  to  use  some  type  of  steepest  descent  technique,  which  tries  new 
values  of  C.  recomputes  q.  and  tiics  to  drive  q to  a minimum.  However,  such  methods  tend 
to  converge  rather  slowly  Also,  numerical  problems  may  ckcui  if  q has  a very  broad 
minimum,  for  round-off  eriors  may  give  rise  to  spurious  local  minima  Instead  of  such  an 
approach,  the  method  described  here  approximates  (4.1  1)  by  a linearization  based  on  the 


ji.mi.tl  oi'Mv  .(!i\ 1 5.  lit  1 , stilvf^  ilic  1 (’'uliiiif'  liiu'i^:'  jiroljlrm,  and  iterates  this  pioccss  to  obtain 
till'  solution  til  till  nonlincai  |jrob!oin  '1  In;  idea  is  similar  to  the  Nc wton-Ra()hsoii  method. 

let  the  II  l>y  m ni.iins.  P be  com|josed  of  the  paitial  derivatives  of  the  functions  in  V. 
such  that 

Of  i 

(4.1.5)  pij  = . 

ayj 

Let  GO  denote  an  a|i|ii oMination  to  G.  Then  equation  (‘1.1.1)  can  be  approximated  as  follows: 

(4.1.0)  U = l(GO)  1 P(CO)*(G  - GO)  + V 

whcie  the  lunciional  drprndence  of  i’  on  G has  been  exjdicitly  indicated.  Define 

(4.1./)  F.  = II  - I ( CO ) 

U = G - GO. 

']  hen  (•!  1 o)  c.iii  be  rewritten  as 

(4 . 1 .B)  (:  r p*|)  + V. 

1 tins  the  nonlinr.ir  equation  (-1  11)  has  been  ri'|il.icrd  by  the  linear  equation  (•!  1.8).  in  which 
L npicsents  the  disciep.mcy  between  the  observations  and  their  computed  values  (usin^',  the 
current  aiiproximations  of  the  parameters),  and  1)  rcpre.sents  the  corrections  needed  to  the 
paiamcteis  • 

It  IS  niiessaiy  to  solve  for  D in  ('1  1$)  in  oiiiei  to  minienizc  q in  (■l.l  ■i).  This  is  a 
standard  pioblem  in  linear  statistical  models  (eg.,  see  [Gi.iybill  71]).  The  solution  for  D is 

(4.1.fj)  D = (P'*WjiP)~*(P‘*W*L) 

and  the  cuvaii.ince  matri.x  of  circus  in  the  solution  to;  D i.s 

(4.1.10)  8 = (I"*W*P)'' 

assuming  that  W is  tlie  invcise  ol  llie  covanance  matii.x  of  the  observation  ertors. 

.Several  othei  qiianiitics  of  iiiteiest  can  be  derived  from  the  solution.  The  expected 
value*  of  q IS  ii-m  If  the  scale  faclur  of  the  covaiiancc  matri.x  of  observation  errois  is 
unknown.  W can  be  adjusted  by  the  raiirj  (n-m)/q  and  S by  the  ratio  q/(n-m).  Otherwise,  q 


c^ii  iisrcl  a<.  ,1  tr<.t  on  tlir  H(i jnMntcnt,  for,  if  the  observation  eriors  have  tlie  Gaussian 
disti ilnitioii,  q (las  the  clii-sqiiaie  clisti ibution  witti  n-ni  degrees  of  freedom  S rc|)resents  tlie 
covaiiancc  matrix  of  mors  m the  adjusted  (larameters  Ttie  square  roots  of  the  diagonal 
elements  of  S aie  the  standard  deviations  of  the  adjusted  parameters.  The  correlation  matrix 
of  tlie  parameteis  can  be  obtained  from  S by  dividine,  the  i.j  element  by  the  product  of  the 
standard  deviations  of  the  ith  and  jth  parameters,  for  all  i and  j 

Giber  results  are  the  covariance  matnx  of  the  adjusted  observations  f’tSiP’  and  the 
cnvaiiance  maiiix  of  the  residuals  \V~  - (P::;S;|:P’).  It  is  often  useful  to  compare  the 
mapiiitudc  of  the  residuals  to  their  standaid  deviations,  i.c,  the  square  roots  of  the  diagonal 
elements  of  tlieir  covariance  matiix  If  a residual  is  greater  than  two  (or  three)  standaid 
deviations  it  indicates  that  the  associated  measured  value  is  inconsistent  with  the  other  values 
used  to  compute  the  estimate  for  the  transform  This  test  is  the  basis  for  the  least-squares 
culling  procedure  mentioned  in  the  previous  chajiter. 

The  covariance  matnx  afiout  a point  not  in  the  solution  is  W'~  ♦ (P;hS:i:P')  where  P is 
the  set  of  jiaitial  dciivatives  at  the  point  and  VV  is  the  inverse  of  the  covariance  matrix  that 
weights  the  measured  values  In  VV  the  standard  deviation  that  can  be  comjiuted  fiom  this 
cnvaiiance  rnaiiix  ran  be  used  to  determine  the  uncertainty  associated  wiih  any  other  [loirit 
on  tfie  object  (eg,  a pctni  oj  intocst).  It  can  also  be  used  to  determine  the  tolctancc  region 
about  the  next  feaiuie  to  be  tried 

The  solution  of  the  iionlincar  problem  can  now  be  described  as  follows.  An  initial 
aji|)rox imation  is  used  to  compuie  the  disci cjiancies  F.i  and  the  partial  derivatives  Pij.  T hen 
f)  is  com|,iiircd  from  (-i  I 9)  and  is  added  to  the  current  apjjroximation  for  C to  obtain  a better 
ajiproximation  T his  piocess  rcjieats  until  there  is  no  further  appreciable  change  in  G Then 
the  filial  values  fiom  the  last  iteiation  can  be  used  to  obtain  S,  Vi,  q,  and  the  other  derived 
quantities  desciibed  above  C)f  course,  in  order  to  converge  to  the  absolute  minimum  of  q 
rather  than  converge  to  some  local  minimum  or  divergence,  it  is  necessary  that  the  initial 
appioximation  be  sufficiently  close  to  the  tiue  solution.  In  most  practical  problems  the  initial 
ap|)ioximatioii  is  not  critical,  in  fact,  often  there  is  only  one  minimum. 

Since  on  the  last  iteration  the  partial  derivatives  have  been  comjiuted  for  the  converged 
value  of  G,  the  solutioii  gives  the  tiue  generalized  least-squares  adjustment  regardless  of  the 
iionlinrarity  llowevei,  some  other  projicrties  of  the  adjustment  arc  only  a|)|iroximate  in  the 
nonlinrai  case  Among  these  aie  the  use  of  S as  the  covariance  matrix  of  the  eriors  in  the 
final  value  of  G,  and  the  projicilies  that  tlie  solution  for  G is  minimum- variance  and 
unbiased  Howevei,  if  the  amount  of  nonlinearity  over  the  lange  of  the  measurement  errors 
IS  small,  these  resulis  will  be  taiily  accurate. 

Fix.uit  -1  1 I is  a flowchart  that  outlines  the  basic  stejis  involved  in  using  a least-squares 
method  to  com|iiite  an  estimate  for  the  location  of  an  object  and  a precision  about  that 
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rstniMlr  I hr  ah  oNihni  a 5.i’i]iii jitial  al?oiithni  ih.u  a(i|)lics  the  leaM-squai cs  loiititic  a<. 
stHUi  a mfliciriit  tmmhri  of  frntuirN  h.i'  hern  loiiiid.  The  best  values  for  the  [(atametris 
aic  used  to  nKi|i  the  object’s  jdaiitird  location  into  an  estimate  for  its  cuiirnt  location  T he 
stariclaicl  cli'viaiicms  associated  with  the  best  paiamcter  values  are  combined  to  firoduce  a 
rroion  of  unceitaiiuy  about  the  estimate  As  stated,  the  algorithm  is  concerned  wiili  the 
location  of  the  object  Given  the  estimate  foi  the  location  of  an  object  and  a |.)recision,  it  is 
easy  to  piodiice  estimates  and  toleiance  legions  for  any  points  of  interest  on  the  object. 


.Section  2 

C:OM  IDI  \GI  IN  UK  I’KFGISION 

'I  he  alv.oiithrn  shown  in  fo  iire  i.l  I can  be  used  by  itself  to  locate  objects  I lowever,  to 
do  so  reinnies  an  assum|itiou  il  the  least- squares  culling  routine  determines  a sti  uctiii  ally 
corisistnit  subset  of  the  ffatiires,  and  if  the  desired  pierisinn  has  been  reached,  then  a coirect 
coi  I es|Kindeiice  h («  been  established  between  the  (lositions  [iioduced  by  the  cipetatuis  and  the 
known  altein  fives  foi  the  te.itiiifs  This  assumption  is  generally  reasonable  when  the 
luimhei  ot  v.nown  altei natives  is  sraall  and  the  ci|rerators  are  reliable  (ic.  they  do  not  locale 
snipiises  vei  y fiequrntly)  llovvcvei.  It  is  possible  to  locate  a set  of  featuies  that  appears  (to 
the  least-squ.iies  ciillitio  routine)  to  be  structurally  consistent,  when  in  fact,  some  of  the  results 
have  hern  incoirectly  assigned  tn  alternatives.  For  c.'iample,  consider  figure  *12  1 Figure 
d I a shows  a point  of  inteiest  and  sets  of  known  alternatives  for  four  operators  Operatois 
niee  and  tour  each  have  two  knov.n  aliei natives.  Figuie  12.1  b shows  the  actual  [Kisitions  of 
.ill  ot  these  points  in  a paiticulai  test  pictiiie  1 hese  positions  ate  the  positions  at  which  the 
opeiators  should  find  thnu,  if  they  aie  leliable  Figure  4 2 I c superimposes  the  best  matching 
(lositions  foi  the  lout  opeiatois  on  top  of  the  actual  |)ositions  If  the  ptogram  decides  that 
operatni  thiee  has  matched  altei  native  3. a and  that  operator  four  has  matched  alternative  4 a 
(both  of  whicfi  ate  wiong),  the  Fast- squaies  routine  will  piobably  decide  that  the  features  are 
s'liictinally  consistent  and  pioceed  to  piedict  that  the  point  of  interest  is  located  at  the 
position  maiked  in  fieine  42  I d 1 his  conclusion  is  wrong.  The  cause  of  tlic  eiror  was  the 
inroiiect  assignment  of  site i natives  to  the  lesults  ot  the  operators,  t he  resulting  assignment 
happtns  to  appear  to  be  stiuctuially  cotisisteut  and  the  program,  having  fooled  itself,  proceeds 
to  diaw  an  imoriect  conclusion  This  c.v.ample  is  a simple  e.vample,  but  it  points  out  a 
pntenti.d  danc.ei  in  unconditionallv  believing  the  results  ol  the  Icast-squaies  culling  routine 

T he  piogpam  m.tkes  the  wiong  irii(>licatic>n  because  (a)  some  of  the  known  alternatives 
aie  incoiiectlv  a'snned  to  the  itsulis  of  the  opeiatois  and  (b)  the  pattern  formed  l>y  the 
incoiieci  assignnunt  happens  to  be  structuially  consistent.  Thus  there  are  two  fiasic  questions 
related  to  tfns  type  of  mistake. 
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(1)  Given  the  value  and  (losition  information  for  several  operators  and 
an  assignment  of  known  alternatives  to  these  results,  what  is  the 
piobability  that  the  assignment  is  correct?  That  is.  what  is  the 
jiiobability  that  all  of  the  individual  assignments  for  the  operators 
are  correct? 

(2)  Given  an  assignment  for  a set  of  0|)erators  that  is  structurally 
consistent,  are  there  other  assignments  for  the  same  results  that  are 
also  structurally  consistent?  1'hat  is.  how  unique  is  the  pattern 
formed  by  the  assignment? 

1 hese  questions  will  be  discussed  in  the  remainder  of  this  section. 

It  is  possible  to  formulate  a version  of  Bayes'  theorem  that  expresses  the  probability 
that  all  of  the  individual  assignments  are  correct,  given  the  position  and  value  information 
produced  by  the  operators  Consider  the  case  of  two  operators.  Assume  that  operator  1 has  M 
known  alternatives  and  opeiatoi  2 has  N known  alternatives.  Let  be  the 

alternatives  for  operator  1 and  gl,g2,  ...citJ  be  the  alternatives  for  operator  2 Then  Bayes’ 
' thcoiem  states: 

1 

(4.2.1)  P[fj.gk  I vl,pl.v2.p2]  = . 

P[vl,v2,pl.p2|n(fj.gk)]  P[-.(fj,gk)] 

1 + * 

P[vl,v2.pl,p2|fj,gk]  P[fj,gk] 

The  piobability  P[  f j , gk | vl , pi  ,v2,p2]  represents  the  probability  that  the  first  operator  has 
located  alternative  f j and  that  the  second  operator  has  located  alternative  gk.  given  the  value 
and  position  information  piuduced  by  the  two  operators  In  other  words,  it  is  the  probability 
that  the  assignment  of  f j to  operator  1 and  gk  to  operator  2 is  correct. 

Formula  4 2 I can  be  rewritten  in  a more  convenient  form  (see  appendix  I): 

(4.2.2)  P[fj,gk  I vl,pl.v2.p2]  = 

1 


P[vl|fr]  P[v2|gx]  P[  pl . p2 , fy . 0^  ] 

*0r  r , s ]* *0[  w,  X ]* 

Ptvllfj] 


1 


V 

1 ♦ ■'(rsj 
& ssk) 


V 

■’(w=j 
Sf  x*k) 


V 

-’(y=j 

& zsk) 


P[v2|gk] 


P[pl.p2.fj.gk) 
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wheic  Ofr.s]  iT|iic5.ciit5  P[  fr  ,gsl -.( f j.gk)  ].  This  version  of  the  formula  is  convenient 
because  the  important  probabilities  form  ratios  that  can  be  easily  evaluated  from  the  density 
functions. 

This  foimula  can  be  extended  to  include  N operators,  but  its  utility  is  limited  by  the 
numbei  of  terms  that  have  to  be  computed.  One  possible  way  around  this  drawback  is  to 
considci  only  the  position  information  produced  by  the  operators.  The  rationale  behind  this 
sim|3lilication  is  that  the  local  value  infoimation  produced  by  an  operator  should  be  used  to 
deternime  the  known  alternative  that  has  been  matched,  if  it  can  do  so  with  the  desired 
confidence.  Otherwise,  combiniiift;  the  value  information  from  two  or  three  operators  is  not 
expccterl  to  acki  any  new  constraints  that  might  distinguish  between  the  possible  alternatives. 
1 he  position  information  of  ati  operator,  howcvei,  is  c.xfiected  to  contribute  new  constraints  if 
It  IS  combined  with  the  position  infoi riiation  from  oilier  operators.  Thus,  if  the  program  is 
tryinp,  to  determine  the  best  assignment  for  two  operators,  it  can  choose  the  assignment 
[f  j.gkj  that  has  the  highest  value  of: 


(4.2.3) 

p[fj.gk  1 

p],p2]. 

Since 

(4.2.4) 

P[fj.{ik  1 pl,p2] 

P[fj.0k,pl,p2] 

P[pl.p2] 

and  the  denominator  in  (‘1.24)  is  the  same  for  all  j's  and  k's,  the  program  can  simply  choose 
the  assignment  that  has  the  highest  value  of  P[  f j.gk.pl  ,p2 ] Once  the  best  assignment  has 
been  found,  the  probability  that  it  is  the  correct  assignment  can  be  computed  by  the  following 
formula 


Even  foimula  4 2 5 involves  a large  number  of  terms  for  a reasonable  number  of 
operators  I oi  example,  ten  operators,  each  with  two  alternatives,  require  1024  probabilities 
to  be  computed  in  order  to  determine  the  best  assignment.  1 his  fact  implies  that  foimula 
4.2.5  should  be  used  to  assign  alternatives  to  small  subsets  of  the  operators  and  then  combine 
these  assignments  to  form  the  overall  assignment.  The  probability  that  the  complete 
assignment  is  correct  can  be  estimated  by  a function  of  the  probabilities  associated  with  the 
subsets.  It  is  even  possible  to  incorporate  some  position  information  for  each  subset.  For 
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example,  the  centroid  of  each  subset  can  be  used  as  the  position  of  the  subset.  Then  the 
overall  probability  that  the  assignment  is  correct  can  be  based  upon  the  structure  of  these 
subsets. 

The  subsets  can  be  set  up  in  advance  or  they  can  be  created  dynamically.  If  the  subsets 
are  created  dynamically  more  and  more  information  can  be  used  to  make  the  assignments,  if 
necessary.  The  idea  is  to  only  use  the  position  information  of  other  operators  if  the  local 
information  is  not  sufficient  to  choose  a known  alternative  with  the  desired  confidence.  The 
assignments  in  section  3.2  were  made  by  choosing  the  known  alternative  that  has  the  highest 
value  of 

(4.2.6)  P[fj  I vl]  (iSjSM), 

This  formula  can  be  extended  to  include  the  position  of  the  match  (see  appendix  II): 

(4.2.7)  P[fj  I vl.pl]  (lijiM). 

If  the  confidence  is  still  not  high  enough  in  the  assignment,  the  position  information  from 
another  operator  can  be  included: 

(4.2.8)  P[fj.gk  1 pl.p2]  (lijsM). 

This  process  can  continue  until  an  assignment  can  be  made  with  the  desired  confidence  or  the 
system  gives  up  because  of  the  amount  of  computation. 

It  is  important  to  choose  subsets  that  are  expected  to  produce  unique  patterns.  For 
example,  consider  figure  4.2.2,  which  shows  the  known  alternatives  for  four  operators.  If  the 
program  is  trying  to  assign  one  of  the  two  alternatives  to  the  results  of  operator  1,  and  more 
information  is  needed  to  distinguish  between  these  two  alternatives,  which  of  the  other 
operators  should  be  used?  Operator  2 may  not  be  helpful  because  it  is  probably  difficult  to 
distinguish  between  the  assignment  [la,2b)  and  [lb,2a].  If  the  initial  constraints  allow  for  a 
small  angular  uncertainty  and  a small  scale  uncertainty,  it  may  be  difficult  to  distinguish 
between  the  assignments  [la,4]  and  [lb,4].  However,  even  if  the  scale  is  allowed  to  change 
fifty  percent  and  the  orientation  in  the  plane  is  completely  unconstrained,  it  is  still  possible  to 
distinguish  between  the  assignments  [la.3a]  and  [lb,3a],  or  between  [la, 3b]  and  [lb,3b]. 
Therefore,  the  subset  should  be  operator  1 and  operator  3. 

Since  the  known  alternatives  for  the  operators  are  known  in  advance  and  the  task 
constraints  are  known  in  advance,  it  is  possible  to  estimate  the  distinctness  patterns  and 
choose  the  best  subsets.  This  choice  can  be  made  at  planning  time. 

The  main  ideas  in  this  section  are  (I)  the  probability  that  an  assignment  is  correct  is 
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important  because  it  determines  wlietlier  or  not  the  results  of  the  least-squares  routine  should 
be  believed,  (2)  the  combination  of  the  position  information  produced  by  several  operators 
adds  constraints  that  help  determine  correct  assignments,  (3)  the  uniqueness  of  a pattern  of 
matches  determines  the  pattern’s  contribution  toward  the  assignment  decision,  and  (4)  the 
uniqueness  of  a pattern  can  be  estimated  at  planning  time  since  the  alternatives  for  the 
operators  are  known  at  planning  time. 


Figure  4.22 
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CHAPTER  5 

PLANNINC-TIME  COMBINATION  RULES 


The  goal  of  this  chapter  is  to  investigate  ways  to  produce  information  that  is  useful  to  a 
strategist.  In  this  context  a strategist  is  a program  (or  possibly  a person)  that  evaluates  the 
various  alternatives  and  develops  a plan  to  achieve  a particular  goal.  At  one  level  a strategist 
might  be  trying  to  decide  whether  to  use  visual  feedback  or  force  feedback  to  check  for  a 
screw  on  the  end  of  the  screwdriver.  At  that  decision  point  it  needs  information  about  the 
expected  costs  and  reliabilities  of  the  alternative  methods  (see  [Taylor  76]  and  [Sproull  77]  for 
descriptions  of  strategists  and  the  information  used  to  make  such  decisions).  This  chapter 
develops  techniques  to  produce  this  type  of  cost  and  reliability  estimates  for  verification 
vision. 

The  execution-time  combination  rules  combine  the  results  of  sequentially  applied 
operators  and  produce  estimates  for  inspection  confidences,  precision,  and  precision 
confidences.  These  methods  make  it  possible  for  the  program  to  stop  gathering  information 
as  soon  as  the  desired  confidence  and  precision  have  been  reached.  The  underlying  program 
structure  is  an  ordered  list  of  operators  to  be  applied.  The  ordering  criteria  are  important 
because  some  operators  are  more  reliable  than  others,  some  operators  contribute  more  than 
others,  and  some  operators  cost  more  to  apply  than  others.  This  chapter  investigates 
techniques  to  rank  the  operators  according  to  their  expected  contributions  and  costs.  It  also 
presents  techniques  to  estimate  the  expected  number  of  features  (and  costs)  required  to  achieve 
certain  confidence  and  precision  limits.  Appendices  IV  and  V demonstrate  the  application  of 
some  of  these  techniques. 
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lUNKING  I'LATURLS  BY  VALUE 


Consiili/i  the  sirtiitlaicl  tajk  of  iiispcciin^;  a scene  to  dccicic  whether  a screw  is  present  or 
not  Section  ?,  I devcl(i|icd  a foininia  that  reduces  the  value  information  from  several 
operatois  into  an  oveiall  confidence  that  the  object  is  present  It  also  pointed  out  that  the 
conttibution  of  an  opei  atoi  is  the  value  of  the  ratio: 


(5.1.1) 


riviioff] 


P[vi lOn] 


where  vi  is  the  value  (or  set  of  values)  returned  by  the  operator  and  On  denotes  the 
proposition  that  the  object  is  present  (see  formula  3.1.12).  For  ranking  purposes  the  logarithm 
of  the  inverse  ratio. 


(5.1.2) 


P[  v1  |0n  ] 


P[vil0ff3 


is  more  convenient.  '1  be  greater  tbc  ratio,  the  better  the  contribution.  The  logarithm  of  the 
likelihood  ratio  is  used  because  it  is  additive  and  there  is  a theorem  (to  be  discussed  in  section 
5 5)  that  uses  it  to  estimate  the  number  of  operators  required  to  reach  a certain  confidence. 

At  pl.anning  time,  vi  does  not  have  a specific  value.  Operator  i has  only  two  density 
functions,  one  for  its  values  when  the  screw  is  On  and  one  for  its  values  when  the  screw  is 
Orr  For  ranking  pinposes  one  is  interested  in  the  expected  value  of  the  log-ratio,  which 
depends  upon  the  total  density  function  for  the  values  associated  with  operator  1.  The  total 
density  function  is  a weighted  sum  of  the  density  functions  for  On  and  Off  (as  shown  in 
figure  3.1.6).  The  weights  arc  simply  the  corresponding  a priori  probabilities.  Therefore, 

(5.1.3)  dcnslty(vl)  = P[On]*On_dcnsity(vi ) + P[0ff ]*0ff_dens1ty(vi ) . 

This  is  a valid  density  function  since 


f dens1ty(X)dX  * f P[On]*On_dcnsity(X)dX  ♦ J P[Off ]*0ff_dens1ty( X)dX 
-to  ~oo 


+00  ♦«» 

» P[On]*Jon_dcnsity(X)dX  P[0ff ]*J'off_dens1ty(X)dX 
-00  -00 


= PtOn]  -0  P[0ff]  * 1. 

The  expected  value  can  then  be  computed  as  follows- 

(5.1.5)  expected_log-ratio  = J log-ratlo(X)  * density(X)  dX. 

-00 

MACSYMA  (see  [Mathlab  Croup  74])  was  used  to  expand  this  integral  symbolically, 
assuming  that  the  density  functions  are  normal.  The  derivation  is  given  in  appendix  III. 
The  result  is  a readily  evaluated  expression  of  the  two  means  (Ml  & M2),  the  two  standard 
deviations  (SOI  Sc  S02),  and  the  a priori  probability  of  On  (i.e.,  P); 

(5.1.6)  exp€Cted_1og-rat1o  = 1og(S02)  - log(SOl)  ♦ 1/2  - P 

2 2 2 2 . 

SOI  ♦ (M2  - Ml)  S02  ♦ (M2  - Ml) 

♦ p, - (1-P)* . 

2 2 
? * S02  2 « SOI 

Later  sections  will  also  need  estimates  for  the  expected  log-ratio,  given  either  On  or  Off. 
The  expected  log-ratio,  given  On,  can  be  computed  as  follows; 


400 

(5,1.7)  ELR_given_On  ■ f log-ratlo(X)  • On_density(X)  dX. 

•00 


The  integral  can  be  expanded  to  produce 
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(5.1.8) 


ELR_givcn_On 


loa(S[)?)  - 1og{S(i]) 


2 2 
SDl  + (M2  - Ml) 

+ — — . 

2 

2 * 5D2 


1 

2 


Similarly,  the  cx|jcctcci  log-ratio,  given  Off,  can  be  cx|)icsspd  as 

2 2 

1 SD2  + (M2  - HI) 

(5.1.9)  ELR_givcn_Off  = log(S02)  - log(SOl)  + . 

2 2 
2 * 501 

Since  the  expected  log-ratio  for  an  0|»rrator  represents  the  0|)erator’s  average 
contribution,  opeiators  that  have  laige  expected  log-ratios  should  be  applied  first  in  order  to 
minimise  the  number  of  operatois  useil  to  reach  a cetiain  confidence  limit.  Thus,  a simple 
operator-ranking  scheme  is  to  compute  the  expected  Ing-ratio  for  each  of  the  operators  and 
then  rank  them  according  to  their  expected  value  (largest  first). 


.Section  2 

KNOWN  ALTF.UNA1  IVFS  AND  SURPRISES 

The  method  used  in  the  last  section  can  also  be  used  to  compute  the  expected 
contributions  for  o|ierators  that  h.ivc  several  kimon  alKmativts  and/or  arc  subject  to 
surprises.  However,  it  is  quite  difficult  to  expand  symbolically  the  integrals  that  express  the 
expected  value.  A numerical  technispie  is  used  instead. 

Formula  3.39  expresses  the  probability  that  the  object  is  present,  given  the  values  of 
several  operators,  each  of  which  may  have  several  known  alternatives  and  surprises.  That 
formula  is 


1 


i 
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(5.2.1) 


1 


P[0n|vl,v2,...vK] 


S P[vjlgj.1]*P[gJ.i] 
P[Off)  K P[On]  OiliNj 

n ( ) 

P[On]  j=l  P[Off)  ^ P[vj|fj.1>P[fj.i] 

OilSMj 


where  fj,l;  fj,2:  ...  fj.Nj  are  the  Nj  known  alternatives  for  jth  operator  when  On  is 
true,  gj.l;  gj,2;  ...  gj.Mj  are  the  Mj  known  alternatives  for  Jth  operator  when  On  is 
false,  f j,0  is  the  surprise  for  jth  operator  when  On  is  true,  and  gj,0  is  the  surprise  for  jth 
operator  when  On  is  false.  The  contribution  of  the  jth  operator  toward  the  overall  probability 
is: 


(5.2.2) 


^ P[vj|gj,1]*P[gj,i] 
P[On]  OSiSNj 

( )• 

P[0ff]  ^ P[vj|fj.1]*P[fj,1] 

OsisMj 


For  ranking  purposes  the  logarithm  of  the  inverse  of  this  ratio  is  used: 


(5.2.3)  log-ratlo(vj)  = log^ 


Z P[vj|fj,1]*P[fj.i] 
P[eff]  OSisMj 


)■ 


P[On]  Y,  Ptvj|gj,i]*P[gj,1] 
OSiSNj 


The  expected  value  can  again  be  computed  by 


+flO 

(5.2.4)  expected_log-ratio  * J 1og>rat1o(X)  * denslty(X)  dX, 


where  the  density  depends  upon  all  of  the  known  alternatives  and  surprises.  Since 

(5.2.5)  P(On]  » P(fj.O)  ♦ P[fj.l]  ♦ ...  PEfj.Nj] 

and  PtOff]  s P[gj,0]  ♦ P[gj,l]  ♦ ...  P[gj.Mj], 


the  density  for  operator  J is 


(5.2.6) 

denslty(X) 


Mj  Nj 

» V(P[fj,1]*density(fj,i))  * ^(P[9j.  1 ]*dens1ty(gj,  1 ) ) . 
isO  1^0 


Thus,  if  ELR  denotes  the  expected  log-ratio  for  the  jth  operator,  then 


(5.2.7) 


ELR 


2 P[vJ|fj.1>P[fj.1] 

^00  P[0f'f]  OstiiMj 

Jlog^  * j * dens1ty(X)dX 

P[0n]  ^ P[vJ|gj,1]*P[gJ,1] 

0Sl5Nj 


or 


(5.2.8)  ELR  « 

^ P[vj|fj.i]*P[fj.1] 

+00 

1og(P[0ff])-1og(P[0n])  + /log^  )*dens1ty(X)dX. 

-00  ^ P[vj|gj,1]*P[gj,1] 

O^I^NJ 


The  logarithms  of  the  sums  could  be  expanded  in  a Taylor  series  in  order  to  integrate  this 
expression  symbolically,  but  it  is  simpler  to  use  a numerical  integration  technique  to 
approximate  the  value  for  a specific  value  of  the  operator.  High-precision  values  are  not 
needed  because  they  are  only  used  to  rank  the  operators  and  predict  the  expected  number  of 
operators  required  to  achieve  a certain  confidence  in  On. 

)t  is  not  necessary  to  integrate  the  function  from  minus  infinity  to  plus  infinity.  As 
discussed  in  section  3.3  each  alternative  defines  an  interval  of  reasonable  values:  plus  or 
minus  three  standard  deviations  about  its  expected  value.  It  is  only  necessary  to  integrate 
over  the  range  that  is  the  union  of  alt  of  the  intervals  for  the  individual  alternatives. 

The  result  of  this  section  is  a set  of  formulas  that  compute  the  expected  contribution  of  ■ 
an  operator,  even  if  it  may  involve  several  known  alternatives  and  surprises.  These  expected 
contributions  will  be  used  in  later  sections  to  compute  other  important  quantities. 
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Section  3 

COST  INFORMATION 

Since  different  operators  cost  different  amounts  to  apply,  a slightly  more  sophisticated 
ranking  scheme  can  rank  the  operators  according  to  a cost-adjusted  version  of  their  expected 
contribution,  i.e.,  the’ expected  value  of 

< log-ratio) 

(5.3.1)  . 

<cost> 

If  it  is  difficult  to  compute  the  expected  value  of  that  ratio,  it  can  be  approximated  by 

Ccxpectcd  log-ratio) 

(5.3.2)  . 

<expected  cost) 

The  cost  of  applying  an  operator  could  involve  such  factors  as  training  time, 
computation  time,  and  memory  space,  but  in  this  discussion,  for  simplicity,  the  expected  cost 
of  an  operator  is  defined  to  be  the  expected  computation  time  required  to  locate  a match. 

Computation  time  is  a function  of  several  variables:  (1)  the  initialization  time,  (2)  the 
‘ number  of  times  the  operator  is  applied,  and  (3)  the  computation  time  for  each  application.  If 

an  operator  is  applied  over  a complete  region  (eg,  the  tolerance  region  about  a known 
alternative),  it  is  relatively  easy  to  predict  the  expected  cost.  However,  if  an  operator  is 
sequentially  applied  in  a region  (using  some  search  strategy)  until  a reasonably  good  match  is 
I found,  one  has  to  predict  the  number  of  separate  applications  to  be  used  to  find  such  a 

match  This  prediction  is  a little  more  difficult.  It  is  based  upon  the  type  of  feature,  the 
expected  distributions  of  the  feature  and  its  alternatives,  and  the  local  characteristics  of  the 
operator  (eg.,  the  size  of  the  region  covered  by  one  application).  Each 
feature-operator-strategy  triple  needs  a separate  mechanism  to  predict  the  average  number  of 
• applications  required  to  find  a match. 

I An  operator-ranking  scheme  that  incorporates  cost  estimates  is:  compute  the 

I benefit-cost  ratios  (as  in  formula  5.3.1)  for  each  of  the  operators  and  order  them  according  to 

I the  largest  first. 
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LEAST-SQUARES  CULLING 

As  nu'iiiioncd  in  section  the  lcnst-si]naics  cnlling  routine  requires  a niininHiin 
iiumUci  of  matches.  Let  M leiJtcsent  this  mintnuim  nunilicr.  Let  N be  the  number  of 
o|)eiatois  lh.it  must  be  .ipplieci  in  orcicr  to  find  M matches.  Since  an  operator  may  or  may  not 
IcKate  a known  altoi  native  (i.c.,  a match),  N is  gieater  than  or  equal  to  M.  This  section 
develops  a method  to  predict  N,  given  M and  an  oidered  list  of  operators.  The  following 
scctioris  derive  methods  to  estimate  the  expicttil  mniiber  oj  operators  required  to  achieve  a 
certain  goal  It  should  be  pointed  out  that  it  is  jmssible  to  estimate  these  numbers  by  simjily 
applying  the  0(ieiatois  to  seveial  tiaining  pictures  and  averaging  the  number  of  operatois 
needed  to  reach  the  desired  goal.  Often  this  direct  way  is  the  best  way  to  proceed.  Llowevet, 
sometinus  it  is  ns>  fnl  to  be  able  to  pioduce  an  liule|jendent  estimate  of  the  expected  number 
of  opciators  1 echrnqucs  to  produce  the.'c  independent  estimates  will  be  discussed  in  the 
following  SfCtions 

In  order  to  piedict  the  average  number  of  o|.ierators  required  to  locate  M matches  it  is 
necessary  to  compute  each  ojieiator’s  e.spcctcd  contribution  toward  M.  Consider  figure  5.‘i.l. 
Figure  5.4  1. a shows  the  possible  matches  associated  with  a typical  operator:  three  known 
alternatives  and  a snrpri.se  (fj,f2,f3,  and  S).  A.ssume  that  the  a priori  probabilities  for 
these  possibilities  are; 

(5.4.1)  P[fl]  = .5. 

P[f2]  = 

P[f3]  = .1. 

.and  P[S]  = .2. 

Figure  5.4  l b shows  the  densities  associated  with  the  various  possibilities,  but  they  are  scaled 
by  their  a piioii  piobabilities  of  occurrence.  Figure  5 4. I.c  shows  the  weighted  density 

function  for  the  operator  That  is, 

« ‘ 

3 

(5.4.2)  dcnslty(X)  = P[S]*deRsUy(S)  + V(  P[  f j ]*dcnsUy(  f j ) ) . 

j=l 

Given  a specific  value  for  the  operator,  the  best  alternative  is  the  alternative  with  the 
highest  probability  of  being  the  correct  match,  i.e.. 


(5.4.3) 


MAX(P[fl|v].  P[f2|v],  P[f3|v].  P[S|v]). 
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1 he  algoiithm  shown  in  figure  3.52  uses  the  probability  associated  with  the  best  match  as  the 
opeiator’s  contribution  toward  the  goal  of  M matches,  except  when  the  operator's  value  is 
unusual  or  it  suggests  that  a surprise  is  the  best  match.  In  the  case  of  an  unusual  value  or 
surprise,  no  contiibution  is  credited  to  the  operator.  Figure  S/l.l.d  superimposes  the  graph  of 
the  operator's  contribution  (scaled  by  a factor  of  10)  on  top  of  the  scaled  densities  shown  in 
figure  5-1  l b Figure  5.‘1  Id  also  labels  each  interval  with  the  name  of  the  possibility  that 
would  be  returned  as  the  best  match.  Notice  that  there  are  three  intervals  that  imply  that  the 
surprise  is  the  best  match. 

The  expected  contribution  of  an  operator  toward  M (abbreviated  EC)  can  be  computed 
in  the  standard  way: 


♦ oo 

(5.4.4)  EC  = J <contribution  at  X>  * density(X)  dX 

“CO 


where 


(5.^.5)  <contr ibot ion  at  X> 


0 (if  unusual  or  surprise) 

HAX (P[fl|X),...P(fnlX])  (o ther u i se) . 


Again  a numerical  integration  technique  is  the  easiest  way  to  compute  the  value  of  EC 

Formula  51-1  is  important  because  it  computes  the  expected  contribution  of  an 
operatoi  Given  an  ordered  list  of  operators  and  their  expected  contributions  it  is  possible  to 
estimate  the  number  of  operators  that  have  to  be  applied  in  order  to  locate  M matches.  The 
expected  number  of  operators  is  the  minimum  N such  that 


N 

(5.4.6),''  ^/operator  j's  expected  contr fbut1on>  ^ M. 

J=1 

This  value  of  N is  the  planning  stage's  prediction  of  the  number  of  operators  to  be 
needed  at  execution  time.  At  execution  time  the  actual  contributions  of  the  operators  can  be 
, used  to  decide  when  a sufficient  i. umber  of  matches  have  been  found.  For  example,  if  the 

I theoretical  minimum  number  of  matches  required  by  the  least-squares  routine  is  two,  then  M 

i IS  two  If  the  planning  stage  predicts  that  each  of  the  operator/feattire  pairs  will  contribute  .5 

j . toward  the  required  number  of  reliable  matches,  N would  be  four  At  execution  time,  when 

I the  operators  arc  applied  to  a test  picttiic,  if  each  operator  happens  to  locate  a best  match 

i _ with  a probability  of  .75,  then  the  first  three  operators  are  sufficient.  If  the  user  wants  to  be 


u 
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pal  ticiilai  ly  consci  vativc  aliuiit  iiicluciinj.  the  smictural  consistency  information,  he  can 
inn  case  tfic  nniiirniini  minibcr,  M,  from  two  to  three.  This  increase  forces  the  [jtogram  to 
apply  nioie  opcratois  befoie  incorporating  the  position  information,  which  means  that  there  is 
a -smallei  chanre  that  the  least -squares  routine  will  piucluce  an  incorrect  precision  estimate  (as 
discussed  in  section  4 2) 


Section  5 
INSPECTION 

In  an  iiisprcticm  task  each  operator  contributes  a certain  amount  toward  increasing  (oi 
clecie.ising)  tli<  ovi  iall  cunficience  that  the  iirtP|josiiion  On  is  true.  Sections  S I and  S2 
developed  techniques  to  compute  the  fxl'cctcil  contribution  of  an  operator.  Given  the 
c\|iectiil  log-iatio  (ie.,  the  expected  contribution)  of  each  opeiator,  what  is  the  expected 
miinbei  of  opi'iatois  lequirocl  to  gnifiatc  a certain  confidence  in  On?  The  answer  to  this 
question  is  based  upon  a theorem  in  sequential  pattern  recognition  [Andrews  72]; 

TIILOKfM:  Let  c(On)  be  the  error  rate  allowed  for  saying  that  On  is 
true  when  it  really  is  false  and  lot  e(Off)  be  the  error  rate 
allowed  for  incorrectly  saying  that  Off  is  true  when  it  really  is 
false.  Let 


1 - e(Off)  c(Off) 

A = and  B = . 

c(0n)  1 - e(On) 

Ificn,  given  that  On  is  true,  the  expected  number  of  operators  to 
he  used  to  make  a decision  is  given  by 

( l-c(0rf  ))nlocj(A)  + e(0ff)*1og(B) 

cxpcctcd_#(0n ) = . 

(average  log-ratio,  given  0n> 

And  given  that  Off  is  true,  the  expected  number  of  operators  to 
be  used  to  make  a decision  is  given  by 

c(0n)*log(A)  ♦ ( l-e(On ) )*log(B) 

cxpcctcd_#(-.Oii)  * . 

(average  log-ratio,  given  Off> 
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And  finally,  the  expected  number  of  operators  to  achieve  the 
specified  error  rates  is 

cxpectcd_#  = P(0n)*expectcd_#(0n)  + P(Off )*expect€d_#(Off ) . 

The  theorem  is  based  upon  the  assumption  that  there  are  an  infinite  number  of 
operators  whose  average  log-ratios  are  known.  However,  there  are  only  a finite  number  of 
operators  (usually  on  the  order  of  ten)  for  any  specific  VV  task.  The  theorem  can  still  be 
used  to  produce  an  approximate  number  of  operators  expected  by  assuming  that  there  are  an 
infinite  number  of  operators  with  the  contribution  of  the  best  operator.  If  that  were  the  case, 
how  many  operators  would  be  needed?  If  the  answer  is  one  or  less,  then  the  best  operator  will 
probably  be  sufficient,  on  the  aver.age.  If  the  answer  is  more  than  one,  consider  the  average 
of  the  fust  two  operators  and  compute  the  number  needed  if  there  were  an  infinite  number  of 
operators  with  that  expected  ratio.  If  the  answer  is  less  than  or  equal  to  two,  the  best  two 
operators  will  be  enough  on  the  average,  etc.  Figure  5.5.1  lists  eight  operators  and  their 
expected  log  ratios.  Using  those  operators  and  a goal  of  e(On)  s e(-iOn)  = .05,  the 
expected  number  of  operators  would  be  one  The  expected  number  of  operators  to  achieve 
e(0n)  = c(-.0n)  = .005  would  be  three. 
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Expected  Log-likelihood  Ratios 
Figure  5.5.1 

1 his  theorem  is  powerful  because  it  provides  a way  to  predict  the  number  of  features, 
on  the  average,  that  will  be  necessary  to  achieve  a specific  confidence.  The  theorem  applies 
to  all  operators  whether  or  not  they  have  several  known  alternatives  and/or  surprises.  The 
only  effect  of  alternatives  and  surprises  is  to  reduce  the  expected  contribution  of  the  operator. 
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See  I ion  C 

F'RECISION 

C.l>a|)tci  •)  dcvcli>|H’ci  a mctliod  to  locate  objects  (in  the  domain  of  VV).  The  method 
divided  a IciLaiion  task  into  thiec  subtasks: 

(I)  locaii.'  eiic.tif.,h  featiiics  to  be  able  to  apply  the  least-squares 
cullin;’,  routine  (tliis  sc:  of  features  is  rcfciied  to  as  the  kernel), 

('?.)  locate  cnouoh  additional  fcatuics  to  produce  the  desired 
precision  about  tire  point  of  interest, 

and  (?)  locate  enough  additional  features  to  develop  the  required 
amount  of  confidence  in  the  statement  of  precision. 

In  order  to  predict  the  total  number  of  operators  needed  in  a location  task,  one  needs 
estimates  for  each  of  the  subtasks.  Section  h.‘I  developed  a method  to  predict  the  c\|)ectod 
luirnbei  of  ojieraiors  lequireci  in  sul'task  (1).  T his  section  and  the  next  section  will  develop 
methods  to  pi  edict  the  expected  number  of  operators  required  by  subtasks  (2)  and  (?), 
rcsiiectively 

Tlicip  IS  .a  precision  associated  wiili  each  opirrator/featuie  pair.  For  example,  given  an 
edge  operator  and  a sjiecific  line  to  be  found,  the  edge  operator  will  locate  a point  on  the  line 
w’ithin  some  piecision  Given  a ciiftercnt  line  (maybe,  a fuzzy  line),  the  precision  of  the  same 
edge  0|>crator  will  probably  be  different.  In  fact,  the  precision  of  most  operators  also  depends 
upon  the  fyl'i'  and  of  change  belwe.n  the  |.ilarining  picture  and  the  test  picture  (eg., 

the  amount  of  rotation  or  the  ch,ingc  in  the  overall  light  level).  In  order  to  predict  the 
number  of  op.-iatois  iiffcled  to  locate  a point  of  interest  within  a piedeclared  precision,  it  is 
necessary  to  model  tl.e  precision  of  c,ich  operator.  A statistical  model  is  convenient  because  it 
jrrovides  vaiiarice  infoi rnation  for  each  piece  of  position  information.  Given  the  variances 
about  tlic  position  information,  the  weight  matrix  (i.e.,  W)  can  be  constructed  The 
least-sipiares  routine  uses  the  weight  matrix  to  determine  the  variances  about  the  resulting 
parameter  v.alucs  and  the  points  of  interest. 

1 he  mcKlelling  of  the  position  information  is  simplified  in  VV  because  there  are  no 
laige  unknown  changes  between  the  planning  picture  and  the  test  picture.  The  variation  in 
the  appearance  of  a feHturo  is  limited.  '1  he  main  factors  that  affect  the  precision  of  an 
operator  aie  (1)  the  inherent  opeiator  characteristics  (e.g.,  its  maximum  resolution).  (2)  the 
Itxal  ^catuie  characteiistics  (eg.,  fuzziness),  and  (?)  small  rotations  (eg.,  15  degrees).  Often  the 
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mlicicnt  chaiactciistjcs  of  an  operator  are  the  doniinant  factors  involved  in  determining  the 
precision  of  the  o|»erator.  If  that  is  true,  an  a prioti  estimate  can  be  used  to  model  the 
precision.  If  that  is  not  true  or  if  a better  estimate  is  desired,  it  is  possible  to  apply  the 
operator  (in  conjunction  with  several  other  ojjerators  with  known  precisions)  to  several  trial 
pictures  and  pioducc  an  estimate  for  the  precision  of  the  operator/feature  pair. 

Given  a statistical  model  for  each  operator/feature  pair,  approximately  how  many 
operators  will  be  needed  by  the  least-squares  routine  to  produce  the  desired  precision?  A 
property  of  the  least-squares  routine  makes  it  possible  to  answer  this  question  in  a 
straightforwaid  way.  If  the  least-squares  routine  is  applied  to  a set  of  N operator/feature 
results,  it  produces  essentially  the  same  precision  no  matter  what  the  actual  position  values 
are.  as  long  as  they  conform  to  the  stated  variances.  Therefore,  if  the  routine  is  applied  to 
the  po.'ition  information  produced  by  N o|ierator/feature  pairs  when  they  are  applied  to  two 
different  trial  pictuies,  it  will  return  approximately  the  same  precision  about  the  point  of 
interest  The  piecisions  are  approximately  equal  because  they  mainly  depend  upon  the 
relative  structure  of  the  features  and  the  expected  variances,  both  of  which  remain  the  same 
from  trial  to  trial  Ihis  property  is  the  basis  for  a procedure  to  predict  the  number  of 
features  needed  to  reach  a certain  precision  given  a trial  picture,  lixate  a kernel  set  of 
matches,  and  apply  the  least-squares  technique,  if  the  resulting  precision  is  sufficient,  stop 
and  return  the  number  of  operators  used  as  the  expected  number  operators  to  be  needed,  if 
the  precision  is  not  sufficient,  locate  another  match,  apply  the  least-squares  routine,  and 
repeat  the  precision  check. 

This  procedure  estimates  the  number  of  operators  required  to  produce  a certain 
precision.  This  estimate  is  an  important  piece  of  information  for  a strategist  If  the 
least-squares  routine  requires  a minimum  of  two  matches  before  it  can  be  applied,  and  if  the 
expected  number  of  matches  required  to  reach  the  desired  precision  is  six.  a strategist  has  to 
cide  when  to  apply  the  least-squaies  routine  One  strategy  is  to  locate  six  matches  and  then 
apply  the  routine  Another  strategy  is  to  locate  two  matches,  apply  the  routine,  use  the  results 
to  reduce  the  searching  required  to  find  the  third  match,  find  the  thir  < match,  apply  the 
routine,  etc  The  second  strategy  may  be  more  efficient  in  terms  of  the  number  of  operators 
used,  but  less  efficient  in  terms  of  the  amount  of  computation  time.  It  may  be  able  to  stop 
after  four  matches,  but  each  application  of  the  least-squares  routine  requires  a certain  amount 
of  com|)utation  time.  It  is  not  clear  which  strategy  is  better.  Sproull  has  investigated  this  type 
of  problem  and  has  implemented  a strategist  based  upon  decision  theory  that  can  decide 
which  strategy  is  better  (Spioull  77].  This  chapter  mainly  develops  techniques  to  estimate  the 
quantities  that  are  of  interest  to  a strategist  like  his. 
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C'.ONI-llU  NCr  IN  'I  III'  I'KF.CISION 

As  irtii.tioiiod  ill  srctioii  •!  I.’  it  is  ciiirn  iT.isun.ililc’  to  assuriie  that  the  location  estimate 
foi  an  object  and  tiie  inecision  alioiii  that  tsiifn.iie  |iiuciiiced  by  the  least-squares  roiitiiu’  is 
collect  Uiidri  that  assiim|irioii,  the  c.\[icctecl  nuniher  ot  ojjcratois  required  foi  a location 
task  IS  the  same  as  the  c.\|;ectecl  number  oi  o|je;alois  needed  to  reach  the  desired  |jrecisioii  It 
tlic  assumption  is  not  true,  it  is  jiossiblc  to  tne  a rueihod  similar  to  the  one  used  in  section  53 
to  estimate  the  expected  number  of  operators  required  to  reach  a certain  confidence  in  an 
ins|ieciion  task. 

Foimula  '12  7 indicates  what  the  contribution  of  an  operator  is  toward  the  overall 
coiificieiice  in  a precision.  Given  this  symbolic  expression  for  the  contribution,  it  is  possible  to 
employ  a niimciical  integiation  rontinc  to  crim|nite  the  e.v/iecfci/  contribution  from  an 
oiaerator.  The  sequential  pattern  recofqiitiori  throiem  referred  to  in  section  5.5  can  be  applied 
again.  Given  the  expected  contributions  for  the  individual  operators,  the  theorem  produces 
the  exfiected  number  of  operators  needed  to  reach  a cei  tain  confidence  in  a precision. 

Given  this  technique,  the  general  prediction  scheme  for  location  tasks  can  be  stated  as 
follows:  di'teiminc  the  expected  number  of  0|ieratois  lequiicd  to  achieve  the  desired 

precision,  determine  the  expected  number  of  operators  required  to  reach  the  desired 
cuiificlence  in  that  piccision  and  return  that  number  as  the  expected  number  of  ojjerators 
required  to  accomplish  the  task.  Tor  cxam|)lc,  il  the  e.xpcctcd  number  of  operatois  required 
to  achieve  tin’  (uecision  is  fcmi,  and  if  the  expected  number  required  to  develop  the 
confidence  is  six  (i  e.  an  additional  two  o|iciator.s  are  needed),  then  predict  that  six  operators 
will  be  needed  for  the  task. 


Section  8 

I’.Xl'KC'l  Kl)  C;OST 

Given  an  ordeicd  list  of  operatois  and  the  expected  number  of  operators  required  to 
achieve  a ceitain  goal  (i.e.,  N),  it  is  easy  to  produce  an  estimate  for  the  expected  cost  associated  ; 

with  achieving  the  goal:  sum  the  expected  costs  for  the  first  N operators,  i.e.,  ' I 

1 

.! 
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(5.8.1)  ^ <cxpected  cost  of  operator  j>. 

J=1 

This  expression  is  just  a rough  estimate  for  the  expected  cost  because  it  assumes  that  the 
expected  cost  is  the  sum  of  the  expected  costs  for  the  expected  number  of  operators,  which  is 
not  generally  true.  A better  estimate  is: 

00 

(5.8.2)  y(P[AJ]  * CJ), 

J=0 

‘ where  AJ  means  that  the  goal  is  achieved  after  applying  operators  one  through  J and  Cj 

denotes  the  expected  cost  of  applying  the  first  j operators. 
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CHAPTER  6 

PROCRAMMINC.  TIME  AND  TRAINING  TIME 


There  aie  four  siaf.es  in  the  development  of  a VV  program;  programming,  training, 
planning,  and  execution  (sec  figure  6 1).  There  are  also  four  types  of  information  that  are 
passed  from  one  stage  to  the  next: 

(1)  A statement  of  the  task,  i.e,  the  feature  of  interest,  a set  of  constraints 
on  the  objects  in  the  scene,  and  the  goal. 

(2)  A camcia  calibiation,  i.e.  the  ti  ansformation  between  the  camera’s 
screen  coordinate  system  and  the  woikstation  coordinate  system. 

(3)  A list  of  operator/feature  pairs.  Each  pair  contains  a feature  that  is 
part  of  one  of  the  objects  in  the  scene  and  an  operator  that  can 
locate  the  image  of  tlic  feature  in  a picture  of  the  scene. 

(*1)  A set  of  statistics  for  the  operators,  ie  statistics  that  describe  the 
reliability  of  the  operators  and  the  distribution  of  values  produced 
by  the  operators. 

The  execution-time  and  planning-time  chapters  described  the  combination  rules,  assuming 
that  all  of  this  information  already  existed  This  chapter  describes  how  this  information  is 
prcxluced  Each  type  of  information  will  be  discussed  in  a separate  section.  The  discussions 
are  based  U|Jon  several  example  VV  tasks.  Annotated  traces  for  two  of  these  tasks  can  be 
found  in  appendices  IV  ami  V The  traces  show  the  capabilities  of  the  current 
implementation  of  the  system. 
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Programming  time 


Statement  of  ttie  task 
Camera  Calibration 

List  of  potential  operator/f eature  pairs 


Camera  Ca I i brat i on 

List  of  potential  operator/feature  pairs 
and 

Set  of  statistics  for  the  operators 


Planning  time 

Statement  of  the  task 
Camera  Calibration 
Set  of  statistics  for  the  operators 
and 

Ordered  list  of  operator/feature  pairs 


PICTURES  OF 
THE  SCENE 


A DECISION 
OR  LOCATION 


Figure  6.1 


Execution  time 
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St'clioii  1 

STATi:  THE  TASK 

I.  An  Inspi'Clim  Task 

1 tic  til  SI  step  III  the  desci  iption  of  a task  is  the  statement  of  the  type  of  task:  inspection 
or  location  For  example,  considei  the  task  of  checking  for  a screw  on  the  end  of  the 
sciewdrivci  'I  he  ultimate  goal  is  a ye:  or  ro:  Is  there  a screw  on  the  end  of  the  screwdriver? 
This  goal  makes  the  task  an  inspection  task.  1 he  remainder  of  the  description  dcjrends  U|jon 
this  fact.  1 he  description  of  an  insjjcction  task  includes  the  following  information: 

(1)  the  (7  ptiori  probability  the t the  object  is  present, 

(?)  the  thresholds  to  bo  reached  in  order  to  decide  that 
the  object  is  iiresent  or  not, 

(3)  the  models  of  the  objects  in  the  scene, 

('t)  the  expected  locations  of  the  objects  in  the  scene, 

and  (b)  the  set  of  constraints  on  the  deviations  from  the 
expected  locations  of  the  objects. 

The  fust  poiiion  ot  the  dialog  in  ajipcndi.v  IV'  shows  how  the  user  currently  specifies  this 
information 

In  Older  to  set  the  a ptiori  piobability  in  the  screw-checking  task,  the  user  has  to 
decide  fiom  fits  cxpeiience  what  percentage  of  the  time  the  Kiewdriver  successfully  picks  up  a 
sciew  from  the  dispensei.  Does  it  pick  up  a screw  nine  times  out  of  ten  or  only  five  times  out 
of  ten?  The  lower  the  a ptiori  , -obability,  the  more  information  the  program  has  to  gather 
to  decide  that  the  screw  is  present. 

The  second  pait  of  an  inspection  task  description  is  a statement  of  the  desired 
thicsholds.  In  oirlri  to  decide  that  the  screw  is  picsent,  the  program  has  to  raise  the 
probability  that  the  sciesv  is  prertni  above  a criiain  threshold.  To  decide  that  the  screw  is 
not  (iicscnt  the  system  has  to  lower  the  piobahility  that  the  screw  is  present  below  a certain 
threshold.  7 he  first  threshold  will  be  referred  to  as  the  yes  threshold  and  the  second  will  bo 
referred  to  as  the  no  threshold.  7 he  user  has  to  detcimine  the  expected  consequences  of  a 
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wrong  decision  and  set  the  thresholds  arcordingly,  taking  into  account  the  fact  that  more 
stiingent  thiesholds  are  haider  to  achieve  For  example,  if  the  arm  jams  the  screw  dispenser 
if  it  tries  to  pick  up  a screw  when  it  already  has  one.  the  no  threshold  should  be  set  to  a low 
value,  eg.  .0005.  This  value  would  force  the  VV  system  to  be  quite  sure  that  a screw  is  not 
on  the  end  before  trying  to  pick  up  another  one 

Sometimes  these  thresholds  will  be  referred  to  as  the  confidences  to  be  reached  by  the 
system.  In  these  terms  the  system  increases  its  confidence  in  a no  decision  as  it  lowers  the 
probability  that  the  object  is  present 

The  third  part  of  a description  is  a set  of  object  models.  An  object  model  is  used  to 
state  the  set  of  constraints  that  limit  the  possible  locations  of  the  object  and  to  describe  the 
structure  of  the  features  that  the  system  can  use  to  locate  the  object. 

1 he  current  object  models  are  simply  a three-dimensional  reference  coordinate  system 
and  a structuied  set  of  features  defined  in  terms  of  that  coordinate  system.  Features  are 
added  as  they  are  needed  to  describe  constraints  or  as  they  are  discovered  during  the 
programming  process  For  example,  in  the  screw-checking  task,  the  screw  is  initially 
represented  as  a coordinate  system  whose  origin  is  located  at  the  tip  of  the  screw.  As  features 
of  the  screw,  such  as  the  corner  formed  by  the  head  of  the  screw  and  side  of  the  shaft,  are 
investigated  they  are  added  to  the  model. 

The  fourth  part  of  a task  description  is  a list  of  the  expected  locations  for  all  of  the 
objects  of  interest  in  the  scene.  For  the  screw-checking  task  this  simply  consists  of  an 
expected  position  and  an  expected  orientation  of  the  screw: 

(6.1.1)  The  expected  position  of  the  screw  (i.e.,  the  tip  of  the 
screw)  is  (60  cm,  45  cm,  5 cm). 

The  expected  orientation  of  the  screw  is  R0T(X,  180*OEG), 
i.e.,  vertical,  pointing  down. 

The  expression,  ROT(X,  180*DEG),  is  an  AL  expression  for  an  orientation  (see  [Finkel  74]). 
It  means  that  the  screw  coordinate  system  is  formed  by  rotating  the  workstation  coordinate 
system  180  degrees  about  its  X-axis.  Since  the  Z-axis  of  the  workstation  points  up,  the 
Z-axis  of  the  screw  points  down 

The  fifth  and  final  part  of  a task  description  is  a set  of  constraints  that  state  the 
location  restrictions  associated  with  the  objects  in  the  scene.  The  following  statement  is  a 
typical  set  of  constraints  for  the  screw-checking  task: 


(6.1.2)  let  o represent  the  angle  betv/cen  the  z-axis  of  the  screw 
and  the  z-axis  of  the  workstation.  Then  the  maximum 
deviations  from  the  expected  positions  are: 

-1.2  cm  S dx  i 1.2  cm 
-1.2  cm  S dy  < 1.2  cm 
- .8  cn  S dz  £ .8  cm 
-10  deg  £ do  < 10  deg. 

Given  the  location  of  a feature  in  the  screw  cooiclinnte  system,  these  constraints  arc  sufficient 
to  determine  the  three-dimensional  volume  that  rcprcsenls  the  possible  positions  for  the 
feature.  Given  this  volume  and  a camera  calibration,  it  is  possible  to  determine  the  tolerance 
region  of  the  featuie  in  the  screen  coordinate  system. 

T he  constraints  mentioned  above  are  all  stated  directly  in  terms  of  the  location  of  the 
screw  even  thoue.h  they  aie  the  result  of  several  other  uncertainties,  such  as  the  accuracy  of 
the  arm  The  u^er  is  res(.ionsible  for  combining  the  vaiious  in.accuracies  and  producing  the 
final  set  of  coii.sti amts  associated  with  the  location  of  the  scicw.  It  would  be  significantly 
better  if  the  tnei  could  state  the  b.isic  inaccuracies  and  have  the  system  compute  the  resulting 
constraints  on  the  screw.  For  example,  the  screw  is  on  the  end  of  the  screwdriver,  which  is 
held  in  the  hand  of  the  arm.  In  this  airangcment  there  are  three  places  for  uncertainties:  the 
arm  (i  e , the  placement  of  the  hand),  the  linkage  between  the  hand  and  the  screwdriver,  and 
the  linkage  betw'cen  the  end  of  the  sciewdrivcr  and  the  screw.  Typical  expected  locations  and 
constraints  on  these  three  sources  of  uncertainty  are: 

(6.1.3)  Tlic  expected  position  and  orientation  of  the  hand  in  the 

workstcitioii ' s coordinate  system  is  (60  cm,  45  cm,  15  cm) 

and  K01(X,  180*DfG). 

The  expected  position  and  orientation  of  the  screwdriver  in 
tlie  hand's  coordinate  system  is  (0  cm,  0 cm,  8 cm)  and 

ROT(X,  0*D[G), 

The  expected  position  and  orientation  of  the  screw  in  the 

screwdriver's  coordinate  system  is  (0  cm,  0 cm,  2 cm)  and 

K01(X,  0*DtG), 

Let  ol  represent  the  angle  between  the  z-axis  of  the  hand 

and  the  z-axis  of  the  workstation.  Then  the  maximum 

deviations  about  the  location  of  the  hand  due  to  the 

inaccuracies  of  the  arm  are: 
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-.6  cm  s dx  s .6  cm 

-.6  cm  $ dy  5 .6  cm 

- .4  cm  S dz  S .4  cm 

-4  deg  i dal  i 4 deg. 

Let  a2  represent  the  angle  between  the  z-axis  of  the 
screwdriver  and  the  z-axis  of  the  hand.  Then  the  maximum 
deviations  about  the  location  of  the  screwdriver  due  to  the 
Inaccuracies  of  the  grasping  operation  are: 

- . 1 cm  s dx  i . 1 cm 

- . 1 cm  s dy  S .1  cm 

- . 1 cm  S dz  i .1  cm 

-3  deg  S da2  i 3 dog. 

Let  a3  represent  the  angle  between  the  z-axis  of  the  screw 
and  the  z-axis  of  the  screwdriver.  Then  the  maximum 
deviations  about  the  location  of  the  screw  due  to  the 
inaccuracies  of  the  attachment  of  the  screw  to  the 
screwdriver  are; 

- . 1 cm  S dx  s .1  cm 

-.1  cm  s dy  S .1  cm 

- . 1 cms  dz  5 .1  cm 
-3  deg  S da3  i 3 deg. 

A constraint  resolving  system  can  reduce  this  series  of  expected  locations  and  constraints  to  an 
expected  position  for  the  screw  and  a single  set  of  constraints  on  the  screw,  such  as  the  set 
stated  in  (6  1.2).  Taylor’s  constraint  resolving  system  provides  this  type  of  capability  for 
position  deviations  and  small  angie  deviations,  where  small  angles  are  defined  to  be  five 
degrees  or  less  [Taylor  76]. 


2.  A Location  Task 


Consider  the  task  of  locating  the  screw  dispenser,  which  is  sitting  upright  on  the  table. 
It  is  a location  task.  The  goal  of  the  task  is  to  locate  the  pick-up  point  on  the  screw  dispenser 
so  the  arm  can  insert  the  tip  of  the  screwdriver  and  pick  up  a screw.  Since  it  is  a location 
task,  the  statement  of  the  task  consists  of 

(1)  an  object  model  of  the  screw  dispenser, 

(2)  an  expected  location  for  the  screw  dispenser, 
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(3)  tlic  known  constraints  on  the  location  of  the  screw  dispenser, 

' and  (^)  the  desired  constraints  on  the  pick-up  point. 

The  initial  model  foi  the  sciew  dispenser  contains  the  origin  of  the  coordinate  system  and  the 
location  of  the  pick-up  point.  A typical  expected  location  for  the  screw  dispenser  is  (55  cm, 
50  cm,  0 cm)  and  ROT  (Z,  ISChrDEC)  in  the  workstation  coordinate  system. 

Since  the  dispenser  is  known  to  be  sitting  upright  on  the  table,  there  are  only  three 
remaining  degrees  of  freedom  X,  Y,  and  a,  where  a is  a rotation  about  the  z-axis  of  the 
workstation.  Typical  initial  constraints  on  the  dispenser  are: 

(6.1.3)  -1 .0  cm  5 dx  S 1 .0  cm 

-1.0  cm  s dy  S 1.0  cm 
-5  deg  i da  s 5 deg. 

Typical  desiied  constraints  on  the  location  of  the  pick-up  point  are; 

(6.1.4)  - . 1 cm  5 dx  5 . 1 cm 

- . 1 cm  s dy  5 .1  cm. 

Thus,  the  goal  of  this  task  is  to  reduce  the  uncertainties  associated  with  the  screw  dispenser  so 
that  the  location  of  the  pick-up  point  is  known  within  the  desired  tolerances. 

All  of  the  constraints  mentioned  above  assume  that  the  camera  is  positioned  once, 
calibrated,  and  left  in  a fixed  position  for  all  of  the  assemblies.  If  that  is  not  the  case,  the 
constraint  computations  need  to  include  the  uncertainties  associated  with  the  repositioning  of 
the  camera 


Section  2 

POSITION  AND  CALIBRATE  THE  CAMERA 

/.  Sfl  up  for  a Location  Task 

There  are  two  main  considerations  that  enter  into  the  placement  of  the  camera: 

( 


(1)  Can  several  features  on  the  objects  of  interest  be  seen? 
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and  (2)  Is  the  resolution  of  the  picture  high  enough  for  the  program  to 
pioduce  the  desired  precision? 

Both  requirements  have  to  be  met  in  order  for  the  placement  to  be  acceptable. 

Intuitively  the  first  requirement  seems  easy  to  meet:  place  the  objects  in  their  expected 
locations  and  then  manually  move  the  camera  around  until  a sufficient  number  of  features 
are  in  view  However,  even  for  this  requirement  there  are  several  complicating  factors  For 
example,  there  have  to  be  a sufficient  number  of  visible  features  at  all  possible  locations  for 
the  objects,  not  just  their  expected  locations  If  the  uncertainty  about  the  location  of  an 
object  includes  an  unknown  rotation  of  plus  or  minus  ten  degrees,  the  VV  system  needs  to  be 
able  to  see  enough  features  at  all  angles  within  that  twenty-degree  range. 

The  resolution  of  the  picture  is  important  because  (a)  it  may  not  be  possible  to  achieve 
the  desiied  precision  if  the  resolution  is  too  low  and  (b)  the  amount  of  searching  may  be 
excessive  if  the  resolution  is  too  high  Thus,  if  the  resolution  is  properly  adjusted  for  a task, 
the  program  can  lixate  the  feature  of  interest  within  the  desired  precision  without  requiring 
an  unnecessary  amount  of  searching. 

The  VV  system  can  help  the  user  find  a good  location  for  a camera  if  it  is  given 
several  pieces  of  information  in  addition  to  a camera  calibration: 

(1)  the  expected  locations  for  the  objects  in  the  scene. 

(2)  the  known  constraints  on  the  objects, 

(3)  a list  of  potential  features. 

(4)  the  feature  of  interest, 

(5)  the  desired  precision  about  the  feature  of  interest, 

(6)  the  types  of  operator/feature  pairs  to  be  used, 
and  (7)  the  expected  resolution  of  the  operators. 

The  first  three  pieces  of  information  can  be  used  to  determine  the  three-dimensional 
volume  that  should  be  visible  to  the  camera  (sec  the  trace  in  appendix  IV).  The  user  roughly 
points  out  several  potential  features,  indicates  which  objects  they  are  a part  of,  and  the  system 
uses  the  set  of  constraints  to  sweep  out  the  total  volume  that  they  might  appear  in  That 
volume  will  be  referred  to  as  the  tolerance  volume  for  the  task.  It  is  essentially  the  union  of 
the  tolerance  volumes  associated  with  the  individual  features.  For  example,  consider  figure 
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62  1a.  which  shows  a model  of  a screw  on  the  end  of  a model  of  a screwdriver  Figure 
6.2.1  b shows  one  feature  and  its  tolerance  volume.  Figure  6.2  l.c  shows  several  features  and 
their  tolerance  volumes.  Figure  6.2.  I d shows  the  task  tolerance  volume. 

Given  a camera  calibration  and  the  tolerance  volume  of  the  task,  it  is  easy  to  check  if 
the  volume  is  included  in  a picture;  project  the  volume  onto  the  screen  of  the  camera  and  see 
if  that  region  is  contained  in  the  picture.  (The  projection  of  the  tolerance  volume  for  a task 
is  the  tolerance  region  for  the  task)  • 

Thus,  if  the  tolerance  region  of  a task  is  contained  in  the  picture,  the  potential  features 
will  be  visible  unless  they  are  obscured  by  other  objects  in  the  scene.  Currently  the  user  is 
responsible  for  deciding  whether  or  not  objects  in  the  scene  will  obscure  the  potential  features. 
Eventually  the  program  will  be  able  to  simulate  the  movement  of  the  objects  in  a scene  and 
decide  automatically  whether  or  not  the  features  are  visible  throughout  the  full  range  of 
uncertainties  for  the  objects. 

Items  (1),  (2),  and  (4)  can  be  combined  to  produce  the  known  precision  about  the 
feature  of  interest  If  the  initial  constraints  happen  to  imply  that  the  initial  precision  is 
greater  than  the  desired  precision,  there  is  no  need  to  apply  any  operators;  the  feature  is 
already  known  within  the  desired  precision.  In  general,  however,  an  increase  in  precision  is 
desired  and  a sequence  of  operators  has  to  be  applied  to  gather  position  information. 

The  expected  resolution  of  each  operator  is  important  because  it  determines  the 
expected  contribution  of  the  operator  toward  the  overall  precision.  Given  the 
two-dimensional  resolution  of  an  operator  in  the  screen  coordinate  system,  it  is  possible  to 
determine  the  corresponding  resolution  in  the  workstation  coordinate  system.  For  example,  if 
a correlation  operator  can  reliably  locate  a corner  within  one  pixel  and  if  the  height  of  the 
corner  is  known,  it  is  possible  to  convert  the  one-pixel  uncertainty  into  workstation  X and  Y 
uncertainties,  such  as 

(6.2.1)  -.07cm  i i '«-.07cm 

-.12cm  $ Cy  ^ <r.l2cro, 

where  Cx  is  the  operator’s  best  estimate  for  the  X coordinate  of  the  corner  and  Cy  is  the 
operator’s  best  estimate  for  the  Y coordinate  of  the  corner. 

A set  of  these  uncertainties  can  be  used  directly  by  the  least-squares  fitting  routine  to 
determine  the  expected  precision  to  be  produced  by  a reasonable  number  of  operators,  such  as 
five  or  ten.  This  expected  precision  can  then  be  compared  with  the  desired  precision.  If  it  is 
less  than  the  desired  precision  the  resolution  of  the  picture  has  to  be  increased.  If  it  is  much 
greater  than  the  desired  precision,  the  resolution  of  the  picture  can  be  reduced. 
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rif.iiic  6 2 2 shows  an  iterative  method  for  iinding  a camera  location  that  satisfies  both 
the  visible  fe.mire  iei|iiiieineiit  and  the  resolution  requirement.  T he  basic  idea  is  to  place  the 
camera  at  some  reasonable  position,  acljusi  the  focal  length  so  that  the  precision  requirement 
can  be  met.  and  then  locally  adjust  the  position  so  that  enough  features  are  in  view. 

An  improvement  in  this  algorithm  can  be  made  by  incorporating  the  following 
interactive  technique; 

(1)  Intentionally  include  more  of  the  workstation  than  is  probably 
nocessaiy  in  the  first  picture. 

(2)  Use  a split  screen  display;  show  the  first  picture,  which  is  the 
wide-angle  view  of  the  scene,  on  the  left  side  and  show  a picture 
taken  at  another  camera  location,  which  is  hopefully  better  than  the 
fust,  on  the  right  (see  figure  6.2.3.a). 

(?)  Have  the  system  superim(.iose  the  tolerance  region  of  the  task  on  top 
of  the  fust  |jicture.  Since  the  first  picture  was  designed  to  cover 
niuic  of  the  scene  than  is  necessary,  the  tolerance  region  of  the  task 
should  fit  within  the  bounds  of  the  picture  (see  figure  6.2.3.b). 

(4)  After  determining  the  desired  change  in  magnification,  scale  the 
tolerance  region  of  the  task  to  the  desired  size,  and  overlay  it  on  the 
light  side  (sec  figure  6.2.3.c). 

(5)  1 hen  the  u.ser  can  adjust  the  camera,  take  a picture,  display  it,  and 
see  if  the  camera  is  correctly  positioned  (see  figure  6.2.3.d).  The 
camera  is  correctly  positioned  if  the  region  outlined  in  the  left 
picture  is  blown  up  at  least  as  large  as  the  region  outlined  on  the 
right  and  the  region  outlined  on  the  left  is  completely  contained  in 
the  pictuie  on  the  right 

This  technique  simplifies  the  eailier  procedure  because  the  user  can  easily  see  the  effect  of 
changing  the  position  and  focal  length  of  the  camera.  He  can  position  the  camera  by 
adjusting  the  poition  of  the  picture  outlined  on  the  left  until  it  fills  up  the  outline  on  the 
right 


1 his  tuhnique  is  interactive,  not  automatic.  Currently  there  is  no  substitute  for 
experience  when  it  comes  to  deciding  which  camera  positions  arc  reasonable  and  which  arc 
not  Fventually  high-level  strategy  programs  will  be  able  to  suggest  reasonable  camera 
locations  from  models  of  the  task  1 o do  th.it  the  strategy  program  will  have  to  consider 
several  factors 


PFRFORn  THF  INITIAL  CAMFRA  CALIBRATION 


Input  the  knoun  tolerances  on  the  objects  in  the  scene, 
input  the  cli’sirerl  tolerances  on  the  point  of  interest, 
pn*;ition  the  ol'iects  and  camera  at  their  expected  locations, 
take  a picture  for  calibration, 

point  out  features  in  the  picture  and  give  their  3-D  coordinates, 

cal ibrate  the  camera, 

specify  the  3-D  volume  that  contains  potential  features, 
determine  the  regioti  of  the  picture  uhere  potential  features  might  appear, 

and 

specify  the  expected  resolution  of  the  operators  to  be  used 


Use  the  calibration  to  determine 
the  effective  3-D  resolution 
of  the  operators  to  be  used 


Use  the  old  picture  and  the 
operator /feature  pairs  to 
recalibrate  the  camera 


Determine  the  expected  tolerance 
produced  by  several  operators 


Take  a new  picture 


Is  the  expected  tolerance  close  NO 
to  the  desired  tolerance? 


Zoom  the  camera  in  or  out 
to  improve  the  tolerance 


s tfie  desired  region  of  the  picture  in  view? 


Return  success!  Adjust  the  position  of  the  camera 


Figure  6.2.2 
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(a)  the  tolerance  volume  of  the  task, 

(b)  the  resolution  of  the  picture, 

(c)  the  other  objects  in  the  scene, 

(d)  the  path  of  the  arm, 

(e)  the  lighting, 

* and  (f)  the  other  vision  tasks  for  which  the  camera  might  be  used. 

For  example,  if  VV  is  to  be  used  to  check  for  a screw  on  the  end  of  the  screwdriver  as  the 

screwdriver  leaves  the  screw  dispenser,  the  camera  must  be  in  a safe  place  and  must  be  able 
to  see  the  screw  To  be  in  a safe  place  it  must  be  out  of  the  way  of  the  arm.  To  see  the  screw 
it  must  be  above  the  table  and  not  behind  the  dispenser.  In  general,  the  placement  of  the 
camera  is  similar  to  the  object  avoidance  problem.  The  program  has  to  locate  a place  for  the 
camera  such  that  the  line  of  sight  from  (he  camera  to  the  object  of  interest  does  not  pass 
through  any  other  objects  in  the  scene. 

If  the  camera  is  not  movable,  it  has  to  be  out  of  the  way  of  the  arm  for  all  of  the 

motions  in  the  task,  not  just  the  one  or  two  proceeding  the  vision  subtask.  For  example,  if  a 

camera  is  to  be  used  to  locate  a hole,  it  has  to  be  out  of  the  working  area  of  the  arm  when  the 
arm  places  the  part  in  the  vise,  aligns  the  top,  inserts  the  screws,  and  finally  transfers  the  part 
to  its  packing  box. 

The  strategy  program  should  also  consider  the  complete  sequence  of  events  when  it  tries 
to  select  a position  for  the  camera.  If  a camera  can  be  used  for  more  than  one  task,  try  to 
position  it  so  that  it  meets  the  requirements  for  all  of  the  tasks.  For  example,  if  a camera  can 
be  used  to  locate  a hole  and  visually  servo. a screw  into  the  hole,  try  to  position  it  so  that  it 
does  not  have  to  be  moved  between  tasks.  In  general,  try  to  minimize  the  amount  of 
repositioning  of  the  cameras. 

2.  Set  up  for  an  Inspection  Task 

For  an  inspection  task  the  resolution  of  the  picture  is  not  critical  as  long  as  the 
operators  produce  sufficiently  different  results  when  the  object  is  present  than  when  it  is  not. 
For  example,  in  the  Krew-checking  task  if  one  correlation  operator  can  reliably  decide 
whether  the  Kiew  is  present  or  not  when  the  image  of  the  screw  is  only  a few  pixels  high,  the 
resolution  of  the  picture  can  be  quite  low  (see  figure  6.2.4.a).  However,  if  one  operator  is  not 
reliable  enough  or  distinguishing  enough,  the  resolution  has  to  be  increased  so  that  several 
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fcHtiiics  aic  vImOIc.  Kiguic  6.2  i b 5.liovvs  the  scicw  enlarged  so  that  the  corners  can  be 
distinguished  An  even  higher  resolution  would  be  required  to  detect  the  texture  of  the  screw 
threads. 


Section  3 

CIIOOSF,  POTFNTIAL  Ol’ERAl  OR/FEATUR E PAIRS 


Now  does  one  choose  potentially  useful  opci  ator/fcaturc  pairs?  Often  the  apjreai arices 
of  seveial  feaniies  aic  quite  siiriilar.  How  can  one  find  opcratois  that  locate  unique  features 
or  operators  that  only  have  a few  known  alternatives?  As  mentioned  in  chapter  2,  a VV 
system  can  piovide  three  levels  of  assistance  to  help  a user  answer  these  questions:  (1)  a 
convenient  cnviionment  in  which  to  cxnei imerit  with  different  0|)crators,  (2)  a list  of 
suggestions  deiived  iioin  typical  pictuics  of  the  task,  or  (3)  a list  of  suggestions  derived  from 
a model  of  the  task.  1 liese  three  levels  of  assistance  arc  discussed  below. 


I 


I.  A Convenient  E rivii otunent 

One  of  the  basic  features  of  the  current  VV  system  is  a user  interface  that  facilitates 
c.xpei imentation  Such  an  inteiface  is  important  because  the  current  analytic  tools  are  not 
complete  enough  to  predict  the  result  of  applying  an  operator  to  a picture.  The  VV  system 
can  not  automatically  investigate  all  asjiccts  of  an  operator.  A user  has  to  help  the  program 
decide  which  opeiator/featuie  pairs  have  the  most  potential.  To  help  the  system,  the  user  has 
to  expel  iment  with  clifferent  0|:iciatois  and  develop  an  intuition  about  them.  Thus,  empirical 
tecliniques  are  used  to  bridge  the  gaps  that  remain  in  analytic  models.  An  operator  is  a[iplicd 
to  several  tiaining  pirtuies  in  order  to  |rredict  how  it  will  behave  when  applied  to  an 
unknown  picture 

'Figure  6 3.1.1  is  a flowchart  for  one  type  of  interactive  system  A user  of  that  system 
might  pioduce  the  following  pioiocol  as  he  selects  potential  opei aior/feature  pairs: 

(1)  Posiiiun  the  screw  dispcnsei  at  its  expected  location  and  take  a 
|)iciure  to  be  used  as  'he  planning,  picture. 

(2)  r>rsciibe  a coiner  that  has  an  internal  angle  of  approximately  90 
di'giers  and  a contrast  across  the  edges  of  approximately  12  grey 
levels  n he  digitisation  of  an  analog  picture  converts  the  intensity 
at  each  pixel  into  one  of  a finite  number  of  grey  levels.  Pictures  are 
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imially  cligitired  to  16,  K.  0-i,  or  CW  grey  levels). 

(?)  I)i'sciil’jc  a coi  relation  o|Jcratoi  to  find  the  corner. 

(•I)  Use  the  canicia  calibration  and  the  constraint  model  to  [Jioduce  the 
tolerance  icgion  about  the  corner. 

(5)  Indicate  that  the  ojjerator  shotild  locate  tlic  three  best  matches  for 
the  coiner,  show  the  (rosilions  of  the  matches,  and  display  the 
associated  correlation  coefficients 

(0)  Move  the  screw  dis])cnscr  to  one  of  its  typical  positions  and  take  a 
|intiiic  to  be  used  as  a tiial  pictuie. 

(7)  Apply  the  correlation  o|jcrator  throughout  the  tolerance  region. 

(S)  Move  the  screw  dispenser  to  another  one  of  its  typical  positions  and 
take  a pictiiic  to  be  used  as  the  second  tiial  pictuie. 

(9)  Ap()ly  the  correlation  operator  throughout  the  tolerance  region. 

(10)  Since  the  appearance  of  the  corner  changes  significantly  from  one 
picture  to  the  ne.xt,  the  correlation  operator  does  not  always  locate 
the  collect  coiner.  Tiy  a corner-finder  instead  of  the  correlation 
operator.  A corner-finder  is  more  reliable  over  a wider  range  of 
rotations,  but  it  is  moic  expensive  to  apply. 

(1 1)  Appply  the  corner-finder  throughout  the  tolerance  region. 

(12)  Input  the  Inst  trial  picnitc  to  see  if  the  corner-finder  can  locate  the 
correct  feature. 

(1?)  Apply  the  coi  ncr-findei  throughout  the  tolerance  region. 

(1-1)  Nfove  the  sciew  disgicnsci  to  another  one  of  its  ty|)ical  positions  and 
take  a |iictuic  to  be  used  as  the  third  trial  picture. 

(lf>)  Apply  the  coi  ncr-findci  throughout  the  tolerance  region. 
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(16)  Since  the  O|icrator  appears  to  be  coirectly  matching  the  desired 
corner,  arid  the  coi  ner-fmder  and  corner  to  the  list  of  potential 
operator/feature  pairs.  1 he  pair  may  be  discarded  later  during  the 
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planning  stage,  but  so  far  it  looks  like  a potential  source  of 
information. 

This  protocol  demonstrates  the  overall  process  involved  in  choosing  potential  operator/feature 
pairs.  Traces  of  this  process  within  the  current  version  of  the  VV  system  can  be  found  in 
appendices  IV  and  V 

Several  of  the  terms  and  operations  used  in  the  above  protocol  need  to  be  explained 
further.  The  remainder  of  this  section  describes  these  concepts  in  more  detail. 

Planning  picture 

T he  planning  picture  is  a picture  of  the  objects  in  their  expected  locations.  It  is  the 
reference  picture  in  the  sense  that  it  is  used  to  define  the  planning  locations  for  all  of  the 
features.  For  example,  the  planning  location  for  the  90-degree  corner  on  the  screw  disjrenser 
can  be  defined  by  pointing  out  its  position  in  the  planning  picture.  Since  the  model  of  the 
screw  dispenser  contains  the  height  of  the  90-degree  corner,  the  camera  calibration  can  be 
used  to  compute  the  corresponding  point  in  the  workstation  coordinate  system.  At  execution 
time  when  the  corner-finder  determines  the  matching  position  for  the  corner  in  the  picture, 
the  same  calibration  and  height  can  be  used  to  compute  tiie  current  location  for  the  corner. 

Since  relative  location  changes  can  be  more  accurately  computed  than  absolute  locations, 
the  execution-time  program  directly  computes  the  relative  change  from  the  position  of  a 
feature  in  a planning  picture,  its  position  in  the  current  picture,  and  its  planning  location  in 
the  workstation.  This  relative  change  can  be  used  to  adjust  the  destination  of  the  arm. 

Describing  a jeature 


There  are  two  basic  mechanisms  th.at  arc  necessary  to  describe  features:  (1)  a cursor 
with  which  to  point  out  features  and  (2)  a model  within  which  to  store  the  description  of  the 
features.  For  example,  consider  the  90-dcgrcc  corner  that  is  part  of  the  screw  dispenser 
shown  in  figure  6.3. 1.2.  In  order  to  describe  that  coiner  the  user  might  do  the  following: 

(a)  Declare  that  he  is  about  to  describe  a corner. 

(b)  Use  the  cursor  to  point  out  the  vertex  of  the 
corner . 

(c)  State  the  height  of  the  vertex. 

(d)  Indicate  a point  on  one  of  the  edges  that  forms  the 
corner . 
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(o)  Indicate  a point  on  the  second  edge  that  forms  the 
corner . 

This  scvjiieiite  ol  actions  cicscnbcs  the  location  of  the  teatiiic  in  the  planning  picture  and  in 
the  woi  kstatitin  cooiuinate  system.  It  docs  not  chsciibe  the  appearance  of  the  feature.  The 
appearance  is  desciibcd  in  conjunction  with  the  description  of  the  operator  used  to  find  the 
fcatuie  A difleicnt  dcsciiptioii  of  the  appearance  is  made  if  a correlation  operator  is  to  be 
used  than  it  a cot ncr-t inder  is  to  be  used. 
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Diflcirnt  types  of  leatnies  iupnie  diffeicnt  tyjas  of  descriptors  For  example,  a textuie 
has  to  b>’  dt'sciilx'l  in  terms  of  fiiqucncics.  si/cs,  and  shapes,  a region  has  to  be  desciibcd  in 
terms  of  boniiflaiy  sej.;rncnls  oi  aic-as;  and  an  cdg;e  has  to  be  described  in  tetms  of  length, 
orientation,  ainl  clistinctiiess  As  new  le.itures  are  added  to  the  VV  system,  the  appropiiate 
dcscripiois  have  to  be  added  to  the  intei active  system. 

l)fictibin°  on  ny>cn7/e; 

Opciatois  an  geneially  paiameteii/ed;  they  have  diffeient  sires,  different  thresholds, 
and  different  veisions  f hf  usci  has  to  set  the  values  for  these  parameters  in  order  to  begin 
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experimcniiiig  with  an  0|iciatot  For  exam(ilc.  in  order  to  apply  a comer-finder,  such  as 
Aharon  Gill’s  (Gill  72],  the  user  may  have  to  specify  an  estimate  for  the  contrast  across  the 
edges  that  form  the  corner.  Often  estimates  for  this  type  of  quantity  can  be  obt.nned  by 
applying  a less  complicated  operator,  such  as  an  intensity  extraction  operator,  to  one  of  the 
training  pictures 

In  general,  the  parameters  for  an  operator  go  through  three  stages  as  the  VV  program 
is  constructed: 

(1)  they  are  initially  estimated,  possibly  with  the  help  of  other 
operators. 

(2)  they  are  changed  several  times  as  the  user  experiments 
with  different  variations. 

and  (3)  they  are  finally  fixed  after  the  training  session. 

As  operators  are  better  understood  and  modelling  techniques  improve,  this  process  of 
successive  ajiproximation  will  be  shortened. 

Describing  where  an  pperator  sfiould  be  applied 

Given  a feature  and  an  operator  to  locate  the  feature,  the  calibration  can  be  used  to 
determine  the  tolerance  region  about  the  feature,  but  where  should  the  operator  be  applied 
within  the  tolerance  region?  That  is,  what  search  strategy  should  be  used  to  locate  the 
feature?  As  mentioned  in  chapter  2 the  strategy  depends  upon  several  factors:  the  type  of 
feature,  the  size  of  the  operator,  the  size  of  the  tolerance  region,  and  the  distribution  of 
occurrences  within  the  tolerance  region  Each  type  of  feature  needs  its  own  method  to 
determine  the  best  search  technique  for  each  situation 

Figure  6.3. 1.3  shows  an  example  of  a search  strategy  to  be  used  to  find  a point  of  a line 
segment.  The  diagonal  line  is  the  expected  location  of  line  segment  in  question.  The 
rectangle  is  the  tolerance  region  about  the  center  of  the  segment.  The  numbered  dots  indicate 
the  position  and  order  of  the  edge  operator  applications.  The  first  part  of  the  search 
concentrates  on  the  region  close  to  the  planned  position  of  the  center  of  the  segment  because 
the  distribution  of  occurrences  indicates  that  the  probability  of  finding  a match  is  highest 
there.  The  pattern  guarantees  at  least  one  point  on  the  line  segment 


Pa^ic  l?2 


6.3.1 


F iguie  G.3. 1.3 


C flecking  for  poicntial  dfcoys 

1 he  leliahilily  and  expected  coiiti ibiition  ol  an  operator  depend  upon  the  number  oi 
potential  coiilusions  for  the  oirerator  For  example,  if  there  are  three  corners  that  all  appear 
similar  to  the  90-c!ev-,ree  corner,  the  corner-finder  may  sometimes  locate  one  them  and  think 
that  it  has  found  the  exjiecteci  90-def,iee  corner. 

T o check  for  decoys  the  u.sei  c.m  ask  the  operator  to  return  the  best  three  or  four 
matches  for  a feature.  If  the  values  of  the  operator  indicate  that  it  is  difficult  to  distinguish 
between  the  matches,  the  0|>erator  may  be  cliscaided.  If  the  matches  are  similar,  but  the 
operator  seems  to  be  able  to  distinguish  tlic  desired  match  from  the  others,  the  decoys  can  be 
added  to  the  mcKlel  of  the  feature  as  known  nhernativts.  The  training  session  will  gather 
statistics  on  the  known  alternatives  and  decide  whether  or  not  they  can  be  reliably 
distinguished  fioni  the  desired  match. 

2 Suggestions  from  Typical  Pictures 


I 


i 


Givcii  a convenient  environment  for  experimentation,  the  user  still  has  to  generate  his 
own  candidates  for  potential  opei ator/fealuie  pairs.  This  section  describes  an  automatic 


6.3.2 


Pat'.e  133 


method  to  generate  such  candidates.  The  idea  is  to  apply  an  operator  to  a training  picture 
and  locate  visu.ally  distinct  features.  For  example,  a procedure  to  suggest  line  segments  might 
be:  apply  an  edge  operator  to  a picture,  link  up  sets  of  edge  segments  that  form  lines,  and 
suggest  as  candidates  those  lines  that  have  a certain  minimum  length,  contrast,  and 
uniqueness.  Given  a list  of  suggestions,  the  user  can  discard  any  of  them  because  they  are  not 
on  the  objects  of  interest  or  because  they  can  not  be  reliably  found. 

This  type  of  automatic  suggestion  system  is  most  important  when  it  is  difficult  for  the 
user  to  be  part  of  the  feature  selection  process.  For  example,  if  a vehicle  is  navigating  across 
the  martian  surface,  it  would  be  difficult  for  a user  to  point  out  features  of  interest.  It  would 
be  possible,  of  course,  but  it  would  be  better  if  the  vision  system  could  make  its  own 
suggestions.  In  programmable  assembly  the  user  is  currently  part  of  the  programming  process, 
but  an  automatic  suggestion  system  is  still  useful  if  it  makes  reasonable  suggestions.  1'he 
better  the  suggestions,  the  less  work  the  user  has  to  perform.  If  a user  asks  for  suggestions 
and  they  are  all  unreasonable,  he  can  easily  revert  back  to  pointing  out  features  himself. 

Line  segment  suggestions 

The  following  list  of  steps  is  a more  detailed  description  of  a line  segment  suggestion 
procedure: 

(1)  Automatically  set  thresholds  for  the  edge  operator 

(2)  Automatically  apply  the  operator  to  the  whole  picture. 

(3)  Automatically  group  the  resulting  edges  according  to  position,  angle,  and 
contrast. 

(4)  Automatically  fit  lines  through  the  groups. 

(5)  Automatically  analyze  the  resulting  line  segments  and  suggest  those  lines 
that  are  a certain  length. 

The  fact  that  the  suggestions  are  produced  automatically  is  important  because  it  reduces  the 
user’s  work  to  the  subtask  of  selecting  good  line  segments  from  the  list  of  suggestions. 

The  technique  referred  to  in  the  first  step  that  automatically  sets  the  thresholds  for  the 
edge  operator  was  developed  by  Binford  [Binford  75],  The  existence  of  this  technique  is 
important  because  it  avoids  the  educated  guessing  that  has  characterized  the  use  of  edge 
operators  in  the  past.  Biiiford’s  technique  makes  it  possible  for  a consistent  method  to  be 
used  to  set  the  thresholds  at  programming  time,  training  time,  and  execution  time. 
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The  t.Hi  ihat  the  i-anie  o|jei  atoi'  is  a[j|jliecl  at  pro^.i ammiri^  time  to  make  the  suggestion 
and  at  execution  time  to  find  the  segment  insures  that  the  suggestions  are  realistic.  The 
suggestions  ate  not  hy|iothctical,  they  are  fcatuics  that  have  been  found  by  a standard  visual 
o|jeiatoi  T his  concieteness  avoids  fe.ituics  tliat  are  not  distinct,  but  it  also  introduces 
suggestions  that  aic  based  upon  strictly  visu.il  phenomena,  such  as  shadows  and  glares. 

The  idea  of  linking  edge  points  together,  mentioned  in  steps  three  and  four,  is  not  new, 
line  followers  have  been  used  within  blocks  woild  vision  for  some  time.  Recently,  however, 
there  has  been  renewed  interest  in  this  problem  (see  [N'arr  75b],  [Nevatia  75],  and  [Binford 
76b])  T he  new  ap|)iQaches  a|.>|jly  edge  operators  to  a picture  and  then  collect  the  edge 
segments  that  form  larger  structures,  such  as  lines  or  curves. 

1 he  line  segment  suggestion  system  can  be  extended  to  suggest  more  complex  features 
such  as  coiiii’is  and  curves,  but  the  heuristics  rcquiicd  to  decide  which  points  are  related  arc 
much  mote  coin|jlex.  A |jartial  step  in  that  direction  is  to  let  the  user  point  out  two  or  three 
points  on  a cuivc  and  have  the  system  automatically  agiply  the  edge  operator  to  follow  and 
chaiacteiize  the  cuive  that  appears  in  the  picture.  A limited  system  of  this  type  has  been 
implrmenied  lor  the  programmable  assembly  environment  (sec  appendix  VI).  Smooth  curves 
aie  particularly  import.iiit  within  programmable  assembly  because  the  parts  are  often  formed 
by  boiing  and  milling  oiicrations,  winch  pioducc  cylindrical  holes  and  other  surfaces  that 
appeal  as  smooth  curves  in  a pictuie 

Coni/atioii  suy fcsHons 

- . 

Some  operaiors  can  not  be  directly  used  to  locate  suggestions  like  the  edge  operator. 
They  rripiirc  more  inioi maiioii  th.ni  a few  paiamcter  values  before  they  can  be  applied.  A 
correlation  opei  aior  is  a piime  example  It  is  defined  in  terms  of  an  array  of  intensity  values, 
which  air  noiinally  talcn  diiecily  from  a planning  (licture.  For  such  operators  a companion 
opei<^or,  called  an  inleifst  Pfuiatoi  can  be  defined  th.at  locates  portions  of  the  picture  that 
arc  likely  candidates  for  that  type  of  opeiatoi  Once  the  interest  o|jerator  has  located  a likely 
portion  of  tfic  picture,  the  operator  can  be  defined,  in  the  case  of  a correlation  operator,  the 
intensity  values  in  tlic  picture  aie  used  to  define  the  correlation  operator. 

<]^uam  and  Moravec  have  both  designed  and  implemented  interest  operators  that  locate 
promising  po'itions  for  correlation  operators  [<!Hian  74]  and  [Moravee  76].  The  basic  idea  is 
that  the  niosi  cflrctivc  coiiclation  opciatois  arc  based  upon  portions  of  a picture  that  contain 
high  variance  information  along  two  ciiflcrcnt  directions.  For  example,  consider  the 
ninty-degiee  angle  shown  in  figure  6.?.C  l.a  It  h.is  high  variance  information  in  the  vertical 
and  hoiizontal  diicciirrns  because  of  the  sharp  changes  in  intensities  in  those  directions. 
When  that  cm lelaiioii  operator  Hies  to  find  a match  in  another  picture,  such  as  figute 
6 3.21b,  which  contains  a simil.ai  comer,  the  two-dimensional  information  will  clearly 
distinguish  the  best  match.  1 he  coiiclation  o|ioiator  shown  in  figure  6.3.2.I.C.  on  the  other 
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hand,  only  has  high  vanancr  infoimation  along  one  direction  When  it  is  applied  to  another 
picture,  such  as  figure  6. 3.2  I d,  it  will  not  be  able  to  locate  a unique  match  because  all  of  the 
' points  along  the  line  match  equally  well  Thus,  good  correlation  operators  should  be  based 

upon  regions  that  have  high  variance  information  in  two  directions.  The  closer  the  directions 
are  to  being  orthogonal,  the  better  the  operator. 

Gennery  has  developed  a correlation  operator  that  directly  measures  the  precision  of 
each  match,  which  depends  upon  the  two-dimensional  information  comained  in  the  operator 
and  the  trial  picture  (Moravec  76],  His  operator  returns  an  error  ellipse  with  each  match 
The  ellipse  indicates  the  two-dimensional  precision  associated  with  the  match.  For  an 
operator  like  the  one  shown  in  figure  6.3  2 1.C  the  error  ellipses  are  generally  elongated 
because  the  precision  is  poor  along  the  edge,  but  good  perpendicular  to  the  edge. 

Both  CHiam's  and  Moravec's  interest  operators  check  local  areas  of  a picture  for  high 
variance  information  in  two  directions.  Moravec's  operator  accumulates  the  sum  of  the 
squares  of  the  differences  along  four  directions:  vertical,  horizontal,  and  the  two  diagonals.  If 
the  local  area  has  no  directional  inform<tion  or  only  one-directional  information,  one  of  the 
four  sums  will  be  close  to  zero  If  the  local  area  has  two-dimensional  information,  all  of  the 
sums  will  be  gieater  than  zeto.  1 he  r.ae.a'-vnc  of  imevest  for  a local  region  is  the  minimum  of 
the  four  sums  In  this  way  it  the  area  has  two-dimensional  information  the  measure  will  be 
significantly  gieater  than  zcio.  Otherwise,  it  will  be  close  to  zero.  To  distinguish  interesting 
areas  from  uninteresting  areas  the  program  compares  the  interest  values  of  the  areas  with  a 
, threshold.  The  program  sorts  the  potential  suggestions  according  to  their  interest  value  and 

presents  them  to  the  user  as  an  ordered  list  of  suggestions 

Figure  6.3.t.2  shows  four  typical  phanning  pictures  to  which  Moravec’s  interest  operator 
has  been  applied  Figure  6 32.5  shows  the  same  four  pictures  and  the  first  fifteen  suggestions 
made  by  interest  operator  Most  of  the  suggestions  are  reasonable  in  the  sense  that  the  areas 
contain  two-dimensional  infjimation.  However,  some  of  the  suggestions  are  on  objects  that 
not  of  interest  For  example,  the  twelfth  suggestion  in  figure  6.3.2.3.a  is  on  the  screw  dispenser, 
which  remains  the  same  whether  the  screw  is  present  or  not.  Hence,  its  values  will  not 
contribute  much  to  the  final  decision.  If  an  operator  of  this  type  happens  to  be  given  to  the 
training  stage,  the  system  will  accumulate  values  of  the  operator  and  form  two  distributions, 
one  for  the  case  with  the  screw  present  and  one  for  the  case  with  the  screw  missing  The  two 
distributions  will  be  essentially  tlie  same.  Since  they  are  essentially  the  same,  the  planning 
stage  will  give  the  operator/feature  pair  a poor  rating  and  the  operator  will  not  be  used. 

A few  of  the  operators  shown  in  figure  6.3.23  are  based  upon  visual  features  that 
change  as  the  objects  in  the  scene  change  locations  For  example,  the  twelfth  operator  in 
figure  6.32  3 b is  centered  upon  an  angle  foimed  by  the  side  of  the  pencil  sharpener  and  part 
of  the  fixture  Since  the  location  of  the  fixture  is  initially  only  known  within  plus  or  minus 
• ten  degrees,  it  may  rotate  ten  degrees  during  an  actual  assembly.  Since  there  is  a large  depth 
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discontinuity  between  the  side  of  the  pencil  sharpener  and  the  fixture,  the  rotation  will  cause 
the  visual  coiiici  to  slide  along  the  side.  T hus,  even  if  the  correlation  operator  locates  the 
same  visual  corner,  it  will  not  be  locating  the  same  point  on  the  pencil  sharpener. 

Operator  number  seven  in  figure  6.3.2.3b  is  another  example  of  an  operator  that  may 
not  locate  the  same  point  on  the  pencil  sharpener  even  though  it  finds  a good  match  In  this 
case  the  operator  is  based  upon  one  of  the  teeth  in  the  main  gear.  If  the  subassembly  rotates 
ten  degrees  away  from  its  planning  position,  the  operator  will  find  a matching  tooth,  but  it 
will  be  the  wrong  one.  The  user  or  the  current  system  should  discard  suggestions  of  this  type 
because  their  position  information  is  not  reliable.  A more  powerful  system  could  use  the  fact 
that  all  of  the  teeth  form  a circle.  If  a VV  program  finds  one  tooth,  it  has  located  one  point 
on  that  circle 

The  training  system  can  automatically  discard  features  that  produce  unreliable  position 
• information  if  it  uses  the  least-squares  fitting  routine  to  check  the  structural  consistency  of  a 
set  of  matches; 

(a)  After  all  of  the  matches  have  been  made,  use  the 
least-squares  routine  to  determine  the  best  fit  for  all  of  the 
matches. 

(b)  Use  the  best  fit  to  determine  the  residual  error  for  each 
feature. 

(c)  Gather  statistics  on  the  residual  errors  of  the  features. 

and  (d)  Discard  features  that  consistently  have  large  residual  errors. 

This  procedure  works  (and  is  demonstrated  in  appendix  V>,  but  it  would  be  more  esthetically 
pleasing  if  the  program  could  analyze  a model  of  the  scene  and  determine  the  character  of 
each  visual  feature.  The  next  subsection  will  discuss  this  type  of  analysis. 

One  of  the  problems  with  Moravec's  current  interest  operator  is  that  some  of  its 
suggestions  are  not  centered  about  the  portion  of  the  picture  that  contains  the 
two-dimensional  information  For  example,  operator  number  six  in  figure  6.3.2. 3.a  is  not 
centered  on  the  corner  that  provides  the  two-dimensional  information.  This  problem  arises 
because  the  inteiest  operator  is  actually  applied  to  a reduced  version  of  the  picture  instead  of 
the  picture  itself.  This  approach  is  nice  because  it  avoids  some  of  the  high  frequency  noise 
that  otherwise  might  confuse  the  interest  operator,  but  it  also  makes  it  difficult  to  locate 
suggestions  accurately.  T his  slight  problem  will  soon  be  solved  by  applying  the  interest 
operator  to  the  reduced  picture  and  then  refining  the  position  of  the  suggestion  by  applying 
the  operator  to  the  original  picture. 
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Figure  6.3.2.2.a 


Figure  6.3.2  2.b 
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Figure  6.3  2.3.a 


Figure  6.3.2.3  b 
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1 he  iiiuresi  oj>erator  also  siK.gcsts.  a tew  legions  that  do  not  have  two-dimensional 
inioi matiuii,  eg  . see  opciaioi  mniibci  I'irtccn  in  figuie  6 3.2.3.C  and  0|jeiator  six  in  figure 
6 3 2.3d  Since  the  v.nianccs  are  only  computed  at  forty-five-degree  intervals,  edges  at 
approMmately  twenty-two  degrees  apjjcar  to  have  some  variance  along  all  four  directions.  If 
these  variances  are  large  enough,  the  area  may  be  erroneously  suggested  as  a good  place  for  a 
correlation  operator  A partial  solution  to  this  pioblem  is  to  increase  the  number  of  directions 
in  which  the  variance  information  is  computed. 

In  iirogiammahic  assembly,  since  suggestions  arc  only  made  once  for  each  task,  one  can 
tolerate  lelatively  slow  methods  that  suggest  potential  operator/feature  pairs.  The  better  the 
suggestions,  the  less  work  required  of  the  user. 


^ Suggestions  from  a Model 

< 

A model-based  suggestion  pioccdurc  uses  a throc-dimensiorial  model  of  the  scene  to 
pioduce  a synthetic  pictuie  of  the  scene  and  then  analyzes  the  synthetic  picture  to  suggest 
potential  opej  atoi /tratiirc  pairs.  '1  he  advantage  of  such  a procedure  is  that  it  can  associate 
visual  fe.Uures  with  the  pans  of  ihs  objects  thtit  produce  them.  Given  this  correspondence, 
the  system  can  potentially  use  all  of  the  properties  of  the  objects  to  decide  which  visual 
features  are  the  best  candidates  for  a VV  task. 

In  this  discussion  a synthetic  picture  is  a picture  that  is  generated  from  an  analytic 
model  of  the  scene.  A real  picture  Is  a picture  taken  by  a camera.  An  analytic  model  of  a 
scene  may  include: 

(1)  object  models  and  camera  models  that  determine  the 
positions  of  the  features  of  the  objects  in  the  picture, 

(2)  light  models  and  surface  models  that  determine  the  grey 
level  for  each  pixel, 

and  (3)  attachment  models  and  constraint  models  that  determine 
the  toleiance  region  for  each  feature. 

T heie  aie  several  levels  of  complexity  foi  each  of  these  models.  For  example,  the  surface 
models  may  be  planar  patches  or  splincd  quadratic  patches  The  better  the  model,  the  better 
the  suggestions 


In  piogramrnable  assembly,  since  many  of  the  parts  are  manufactured  from  mechanical 


6 3.3 


Page  !•}  1 


drawings,  it  is  relatively  easy  to  piodiice  the  basic  object  models.  Hopefully  as  object 
modelling  is  improved,  a three-dimensional  description  language  will  be  developed  that  can 
be  used  to  control  all  of  the  jirocessing  associated  with  a part:  manufacturing,  inspection,  and 
assembly.  Such  a language  would  help  standardize  parts  and  unify  the  process  of  assembly. 

A fust  step  towaid  an  automatic  suggestion  system  has  been  implemented  It  is  used  to 
suggest  promising  curve  segments.  It  is  based  upon  Miyamoto's  modelling  and  graphics 
system  [Miyamoto  7fi].  which  in  turn  is  based  upon  Binford’s  spine-cross-section  model  (see 
[Agin  72],  [Nevatia  73],  and  [Miyamoto  75]).  Each  part  within  a spine-cross-section  model 
consists  of  a sjiine  and  a sequence  of  smoothly  varying  cross-sections  along  that  spine.  For 
example,  a shaft  can  be  described  as  a straight  line  that  has  circular  cross-sections  (see  figure 
6 3.3  1. a)  A rectangular  bo.\  can  be  described  as  a straight  line  that  has  rectangular 
cross-sections  (see  figure  6.3.3. l b).  Smooth  curves  are  approximated  by  a sequence  of  line 
segments  Tfie  user  sjiccifies  the  curve  and  the  number  of  segments  to  use  to  approximate  the 
curve  and  the  system  produces  the  correct  sequence  of  line  segments.  This  representation  has 
the  nice  pioperiy  that  it  distinguishes  between  lines  that  are  actually  part  of  the  object  and 
lines  that  are  used  to  ajiproximate  a curve  Since  this  distinction  is  made,  the  system  can  be 
smart  about  which  lines  it  displays.  For  e.xample,  the  hidden-line  view  of  the  shaft  shown  in 
figure  6.3  3.1  a docs  not  show  all  of  the  visible  approximating  lines,  only  those  that  outline  the 
part  (see  figure  6.3.3.  l.c).  Figure  6.3.3. l.d  shows  the  standard  hidden-line  view  that  includes 
all  of  the  visible  lines 

The  steps  in  the  curve  segment  suggestion  procedure  are: 

(1)  Interactively  construct  models  for  the  objects  in  the  scene  and  place  them 
at  the  correct  relative  locations  with  respect  to  the  camera. 

(2)  Use  a hidden-line  scheme  to  delete  lines  that  are  not  visible  and  produce 
a two-dimensional  line  drawing  of  the  scene 

(3)  Intel  actively  point  out  promising  curve  segments  that  appear  in  the  line 
drawing. 

(4)  Automatically  fit  a smooth  curve  through  the  broken-line  segments  that 
approximate  the  curve  segment. 

(5)  Interactively  estimate  the  contrast  across  the  curve. 

(6)  Automatically  use  the  smooth  curve  to  locate  and  characterize  a similar 
curve  in  a real  picture  of  the  scene,  if  one  exists. 

(7)  If  the  characterization  of  the  curve  indicates  that  it  has  sufficient 
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ci'iitiH^t,  kiM.th,  iiiicl  <.1110011  III  11. ss,  sold  ilie  ccl.  e o|jcratoi  and  curve  to  the 
oi  (loioiiii.il  0|Jcr.iioi/l(Mtuic  |i,nrs 

Fi;;uie  0 3 32  shows  a linr  ciinwinf  jnodiicrd  tioin  a typical  model  and  a real  picture  of  the 
cui  I csponclnik',  soeno  I In'  object  is  the  points  assembly  on  the  shaft  of  a chainsaw  engine. 
1 he  ust'i  intei  artivriy  points  out  one  of  the  curve  segments  and  the  system  automatically 
chai  actei  ires  it  I n.uie  0 3 3.3.a  shows  one  of  tin.  bioken-line  approximations  overlayed  on 
toj>  of  the  real  pietiiic  1 ig.iirc  6 3 3 30  shows  the  smooth  curve  that  the  system  fit  through 
the  bioken-line  stj.n'ents.  1 igure  G 3,3.3c  sliows  the  initial  applications  of  the  edge  operator 
that  aic  dcsigncil  to  locate  points  on  the  ariual  curve  I igurc  6.3.3.3.d  shows  the  final  curve 
produced  by  the  cuive  followei.  It  extends  from  the  right-angle  corner  at  the  top  to  the 
shadow  at  the  bottom. 

Since  the  piogiain  can  maintain  pointers  that  relate  features  in  the  synthetic  picture  to 
the  ofrji'Ct  models  that  piocluce  the  featuies,  it  is  theoretically  possible  for  the  program  to 
aveud  seveial  ol  the  jiHiblniis  iiii  niinneii  in  the  pn  vioiis  subsection,  tor  cxam|}le,  the  system 
could  deieimine  that  the  iight-nn  le  corner  at  ihi  end  of  the  smooth  curve  is  formed  by  two 
objects  that  h.ivc  a cli  pth  disconiininty  between  tin m;  the  shaft  is  in  front  of  the  side  of  the 
points  a'sembly  ctisiny  Such  a coriioi  would  not  be  a good  suggestion  because  its  position 
and  ajipcaiance  will  ch.mgc  as  the  locations  of  the  ohjrcts  in  the  scene  change. 

1 he  systi  in  could  also  decide  whethci  or  not  a visual  feature  is  foimed  by  a shadow.  If 
a light  mcidel  is  available,  the  sysiem  could  directly  l.ahcl  all  of  the  synthetic  visual  features 
that  aic  foiiued  by  sh.adows  or  glaies  If  a light  model  is  not  available,  the  system  could 
check  the  line  di awing  for  object  features  that  mi  lit  coircsjiond  to  a specific  visual  feature 
If  theie  aie  no  o.bjeci  features  that  might  explain  the  visual  feature,  the  visual  feature  would 
be  classified  as  a tiaiisient  feature  '1  i aiisient  features  include  such  things  as  dirt  sjjots, 
slmdciws,  and  gl.in.s.  1 or  example,  one  end  of  the  smooth  cuive  in  figure  6.3.3.3.d  is  lost  is  a 
shadow  V\'hcn  the  chai actei iraiioii  loutiuc  notices  the  change  in  contrast  and  distinctness  as 
It  IS  following',  the  cuive,  it  could  ask  the  modelling  system  if  the  change  is  due  to  a feature  of 
the  object  or  some  tiaiisicnt.  In  this  c.isc  the  chau.;e  is  due  to  a transient. 

1 his  s.anir  tyjie  of  suggestion  piocrduie  can  suggest  other  features,  such  as  corners,  line 
segments,  and  ngions  I he  sysiem  htis  to  be  aide  to  locate  the  features  in  a line  drawing  of 
the  scene  and  deride  if  the  grey  levels  an.  di.siinct  rimugh  to  make  them  easy  to  find. 

'l  lic  ultimate  suggestion  system  would  h.  aide  to  use  the  constraint  information 
associated  with  the  objects  in  the  scene  to  pcrfoim  a more  comprehensive  analysis.  It  could 
dctciminc  the  niimbri  of  known  alternatives,  the  rarig;e  of  appearances  for  a feature,  and  the 
existence  of  degenei ate  views 
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GATHER  STATISTICS 

J The  7 raining  Session 

Given  a list  of  opei ator/foatnre  pairs,  the  purpose  of  the  training  phase  is  to  gather 
statistics  on  the  reliability  of  the  operators  and  determine  the  distributions  of  values  produced 
by  tfre  operators  '1  his  information  is  passed  to  the  planning  phase  where  it  is  used  to  rank 
the  potential  opeiator/feature  pairs  and  to  the  execution  phase  where  it  is  used  to  determine 
the  likelihood  ratios  that  coirespond  to  the  values  produced  by  the  ojrerators, 

T he  training  phase  is  generally  cunsideied  to  be  a one-shot  learning  process  that 
determines  the  a priori  information  associated  with  the  operators.  In  programmable 
assembly,  however,  since  the  same  task  is  repeated  many  times,  additional  statistics  can  be 
gathered  during  the  execution  phase  The  system  can  dynamically  maintain  models  for  alt  of 
the  opeiators.  Dynamic  models  can  adjust  for  gradual  changes,  such  as  gradual  lighting 
changes 

1 tie  reliability  statistics  aic  based  upon  the  relative  number  of  times  the  operator 
locates  the  coriect  match  Each  time  the  operator  is  applied  there  are  three  possible  outcomes: 

(1)  the  operator  locates  the  corrti.t  match, 

(2)  the  operatoi  locates  one  of  the  known  alternatives. 

01  (?)  theopeiatoi  locates  a suiprise 

If  theie  aie  two  known  aliernatives  in  addition  to  the  coirect  match,  there  are  really  four 
possible  outcomes  1 he  user  supervises  the  training  session  and  indicates  the  outcome  for 
each  a|i|j|ication  of  each  o|jerator  (sec  appendix  IV). 

1 he  system  ines  the  percentage  of  times  that  the  operator  matches  a possibility  as  an 
estimate  for  the  a priori  piobability  of  that  possibility.  For  example,  if  an  operator  matches  a 
sui prise  three  times  out  of  thirty  apjrlications,  the  system's  estimate  for  that  operator's  a priori 
probability  of  a surprise  is  0 I'l. 


I 


1 he  training  phase  also  gathers  statistics  on  the  values  produced  by  each  o|;erator.  If 
an  operator  has  four  possible  outcomes,  the  system  gathers  statistics  to  approximate  the  four 
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cliMi  ibuiRiii!.  It  the  ta'.l  15  an  iii'|icciioii  ta'-k,  the  system  f.athcis  one  set  of  statistics  for  the 
case  III  which  the  ohjert  is  picsriit  and  anothci  set  ot  statistics  for  the  case  in  which  the 
object  IS  nnssiiig. 

I he  type  of  statistics  gathered  clepcncls  upon  the  e.spected  form  of  the  distribution.  If 
the  di5t!  ibiitioii  IS  expected  to  be  a normal  distril.uition,  the  sum  of  the  values  and  the  sum  of 
the  S'.jtiates  of  the  values  are  suiticient  to  determine  the  distiibution  uniquely.  As  mentioned 
in  section  ?G2  theie  is  a change  o:  variable  that  conveits  correlation  values  into  values  that 
fotm  an  appioximaielv  noimal  curve  Thus,  for  coiielation  operators  the  changed  values  are 
used  instead  of  the  cm  relation  coefficients  If  an  ci|)Crator  produces  an  unusual  distribution 
and  if  a scaled  veisiun  ol  the  histogram  is  to  be  used  as  the  distribution  of  the  operator,  the 
system  stores  the  values  themselves. 

biguie  GH  I I is  a ilovvchaii  fur  a tiaiiiing  sr■s^|^.||  based  11(1011  these  ideas.  It  uses  a 
S(dit  screen  to  dis|ilav  the  |il.innin>‘:  (iicttne  on  the  left  and  the  current  training  picture  on  the 
right  In  tins  way  the  (il.niiiiiig  (lositions  and  ajijii  ai ances  for  the  features  can  be  overlayed 
on  top  ot  the  let!  (ucture  and  the  matching  (.uisiiions  ran  be  overlayed  on  top  of  the  right 
j |iicturc  I he  iisei  can  easily  si  i-  v.’lmh  ft...tuies  a^e  being  looked  for.  where  they  are  located, 

and  wliether  01  not  tlie  o|icrator  finds  the  correct  match 

"1  his  (Miticiilar  tiainmg  method  di.s|ilays  an  rx(jcctcd  feature  and  all  of  its  known 
aln  I natives  on  the  Ic  n and  the  mi'clnng  feituic  on  the  light  (see  figure  6.-1. 1.2).  One 
0(icrator  at  .a  time  is  disjil.iyed  and  the  user  decides  which  alternative  has  been  matched  (eg., 
iiurnbri  2 Ml  ligine  G.-i  1 2).  A sejiatate  disjjlay  for  each  operator  is  necessary  if  the  0|jerators 
have  seveial  known  altetnaiivos  1 lowever.  if  only  a few  of  the  features  have  known 
alieinativcs  it  is  mole  efficient  to  show  all  of  thC'  c.x|)ected  features  and  all  of  the  matching 
(lOsitions  at  once  (see  figiiie  C.i  1 ?)  T he  user  can  still  decide  the  outcome  of  each  operator 
and  indicate  it  to  the  sysnm 

If  the  o|»iai(iis  almost  always  locate  the  collect  match,  the  process  can  be  shortened 
fuithei  so  th.it  the  inei  ruily  s|,r.ifies  the  results  for  the  0|)eiators  that  miss  their  ex|iected 
match  (!<e  tin  ti  ace  ni  the  tiaining  |>hase  in  a|jpi.ndix  IV).  1 his  3(j|)roach  makes  it  is  easy  to 
gather  the  ti.iinm  ; inloi iiiarinii:  the  usei  (Misitiriiis  the  objects  at  tyjiical  positions,  takes  a 
pictuie,  apiiliec  the  cipiia'ois  to  the  |jicture,  and  indicates  which  ones  locate  something  other 
th.an  then  c.x|ircied  le.iiiire 

Ciiven  .1  system  that  can  eacily  gathci  the  a)j(iiOjii iatc  statistical  information,  the  user 
still  has  two  coiici  ins  (I;  tin  nuinb<.  1 of  iraiiuug  (uciuies  to  use  and  (2)  the  arrangement  of 
the  objects  to  use  to  tale  the  (iicinies.  I he  set  of  tiaining  |)ictiires  should  include  enough 
pictuics  to  d(.'ii.iiiiine  the  value  disti ibulions  of  the  0|)eiators  and  a sufficient  range  of 
pictures  to  piovidr  the  system  with  a ir|i;rsf  ntativc  set  of  situations 
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Input  the  statement  of  the  task,  the  camera 
calibration,  and  a list  of  operator/feature  pairs 


Input  a planning  picture  and  display 
it  on  the  left  of  a split-screen  display 


Input  a training  picture  and  display  it  on  the  right 


Is  the  task  an  Inspection  task?^ ► Ask  user  if  the  object  is  present 


Display  the  planning  positions 
for  all  of  the  knoiin  alternatives 


Display  the  correct  set  of 
knoun  alternatives,  depending 
upon  whether  or  not 
the  object  is  present 


Apply  the  operator  to  the  feature's  tolerance  region  and 
display  the  matching  position  in  the  training  picture 


Ask  if  the  match  is  the  correct  feature,  one  of 
the  known  alternatives,  or  a surprise 


Add  the  operator's  values  to  the  appropriate  sums 


More  untried  operators?! ►lllore  training  pictures?! ►!  Compute 

, Statistics, 


Figure  6.4. 1.1 
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Figure  6.4  1.3 
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1 he  number  of  training  pictures  depends  upon  the  type  of  the  distribution.  Twenty  to 
fifty  are  normally  required  for  normal  distributions  (see  section  6.4  1.2).  Scaled  histograms 
often  require  over  a hundred  if  the  values  are  widely  distributed. 

It  is  more  difficult  to  specify  which  pictures  should  be  used.  The  overall  distribution  of 
pictures  should  roughly  correspond  to  the  expected  distribution  of  object  arrangements.  The 
more  representative  the  training  pictures  are,  the  better  the  statistics  will  be  The  set  of 
training  pictures  should  include  some  of  the  unusual  and  special  situations  in  addition  to 
several  of  the  standard  ariangements.  For  example,  the  set  should  include  pictures  of  the 
objects  when  they  are  at  their  extreme  locations,  such  as  at  their  maximum  rotations  and 
translations.  These  pictures  insure  that  the  operators  can  locate  their  matches  when  the 
features  are  distorted  the  most  T he  set  should  also  include  any  degenerate  cases  that  involve 
one  or  more  of  the  features.  For  example,  if  a corner  is  one  of  the  expected  features  and  if 
an  edge  on  anotlier  object  happens  to  ajjpear  close  to  the  corner  when  the  objects  are  in  one 
particular  arrangement,  that  arrangement  should  be  included  in  the  training  sequence.  In 
order  to  present  the  correct  relative  importance  of  the  pictures,  all  of  them  should  be  weighted 
by  their  a priori  probability  of  occurrence. 

Given  a sufficient  number  of  training  pictures  that  reflect  the  expected  distribution  of 
scenes,  the  training  system,  under  the  supervision  of  the  user,  produces  behavior  models  of 
the  operators.  The  planning  system  analyzes  these  models  to  construct  a plan  to  achieve  the 
goal  and  the  execution  system  uses  these  models  to  make  decisions  based  upon  the  behavior 
of  the  operators  when  applied  to  the  pictures  of  unknown  situations. 

2.  Number  of  Trials  for  Normally  llislributed  Ualues 

After  deciding  which  distribution  to  use  to  model  the  results  of  an  operator,  one  still 
has  to  decide  how  many  training  pictures  to  use  in  order  to  produce  a good  approximation 
for  the  distribution.  If  the  chosen  distribution  is  normal,  one  needs  enough  samples  to 
approximate  the  mean  and  standard  deviation,  since  a normal  distribution  is  completely 
determined  by  these  two  parameters.  Flow  many  samples  are  needed?  There  are  two 
theorems  that  help  answer  this  question  (sec  [Floel  71]): 

THEOREM:  If  X Is  normally  distributed  with  variance  V and 

U 2 

Z (XI  - Ms) 

1«1 

Vs  = 

N 
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IS  the  Sciinplc  varinnee  based  upon  a random  sample  of  size  N and 
Ms  is  the  sample  mean,  then 


NaVs 


V 

has  a ciii-square  distribution  with  (N-l)  degrees  of  freedom. 

TlltOREM:  If  X is  normally  distributed  with  mean  M and  variance  V and 
a random  sample  of  size  N is  taken,  then  the  sample  mean  Ms  will 
be  nonnally  distributed  with  mean  M and  variance  V/N, 

Let  C5(n,p)  icpicsen't  the  value  such  that  a chi-Si.|uai'e  distribution  with  n degrees  of 
ficcdoni  has  p priccnt  of  the  po|:iuhtion  to  the  right  of  that  value.  One  application  of  the 
first  theorem  states  that  there  is  a ninety-five  percent  chance  that  the  sample  variance  and 
actual  vaiiance  aie  related  as  lollows: 


N*Vs 

(6. 4. 2.1)  CS((N-1), .975)  S S CS( ( N- 1 ) , . 025 ) . 

V 


Let  S and  Ss  represent  the  standaicl  deviation  and  the  sample  standard  deviation  of  the 
distribution.  Since  Vs  « Ss-tSs  and  V = S^rS,  formula  6.4.2. 1 can  be  converted  into  the 
following  statemetit  concerning  the  actual  and  sample  standard  deviations: 


sqrt(N)  * Ss  sqrt(N)  * Ss 

(6. 4. 2. 2) < 5 S . 

sqrt(C.S((N-l ),  .025))  sc)r  t ( CS(  ( N- 1 ) , .975)) 

1 he  secnricl  thecucm  can  be  u’-ed  to  produce  a ninety-five  peicent  confidence  interval 
about  the  mean  That  is, 


2aS 


(6. 4, 2. 3)  |M  - Ms|  i 

sqrt(N) 

or,  substituting  the  larger  value  from  (6  4.22)  into  (6  4,2  3)  produces 

2aSs 


(6, 4. 2. 4)  IM  - Ms|  <;  

sqrt(CS((N-l), .975)) 
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For  example,  if  Ms  - 13  and  Ss  « 2.  the  ninty-five  percent  confidence  intervals  based  upon 
a sample  size  of  1 5 are 

(6. 4. 2. 5)  |M  - Ms|  5 .16950 

and  ,15092  5 5 S .32824. 

For  a sample  size  of  30  the  intervals  arc 

(6. 4. 2. 6)  |M  - M.s|  i .10022 

and  .16151  S S S .27446. 

One  iriteiesting  possibility  is  to  use  the  pl.mning-time  formulas  to  predict  the  effect  of 
gathering  more  s.amplcs  from  an  operator’s  distribution.  Two  important  questions  can  be 
answered  in  this  way: 

(1)  Given  a sample  mean  and  a sample  standard  deviation,  plus 
confidence  inteivals  about  them,  what  is  a reasonable,  but 
ronsftvattve  distribution  (or  set  of  distributions)  that  can  be 
used  to  model  the  operatoi? 

(2)  Given  an  additional  set  of  N samples  from  a distribution,  what 
is  the  probable  change  in  the  operator’s  expected  contribution? 

In  this  situation  a conservative  distribution  is  a distribution  that  understates  the  contribution 
of  the  opeiator.  1 he  use  of  such  a distribution  may  require  more  operators  to  be  applied 
than  theoretically  necessary,  but  there  is  a smaller  chance  of  making  an  incorrect  decision. 
For  example,  assume  that  a potential  0|)Crator  in  an  inspection  task  has  the  following 
characteristics 

(6. 4. 2. 7)  (sanplc  size  of  15) 

On  Off 

Ms  = 1.3  Ms  = 1.95 

Ss  = .2  Ss  = .22 

Assume  that  the  piobability  of  On  is  .9.  Then  the  expected  log-ratio  for  the  operator  is  3.41. 
To  pick  a moie  conservative  distribution  for  the  operator  consider  the  sixty  percent 
confidence  intervals  about  the  means  and  standard  deviations: 


& -i  c 


r.i-e  LSi 


(6.4.?.0)  (Stimplc  s^^c  of  15) 

On  Off 

1.234<  n <1.3Gb  1.8/85  M <2.022 

.1825  S <.252  .2005  S 5.277 

I)  one  3^^.1111105  ili.it  the  the  most  conn.’i  v.itivc  set  oi  ciisti iljtitions  Is  [jroducecl  at  the  extierncs 
ol'  these  Intel  v.ils,  tlieic  are  sixteen  |iossible  ccnvihiii.iiioiis  for  tlic  pair  of  disti  ibutions  to  h. 
iiseil  t(i  nuulel  ih>  upeiaini.  1 1;  tile  0 -i  f?  1 shows  the  expected  contribiitioii  of  the  o(jei  atoi  loi 
e.uh  ol  tlic  sixteen  possibilities  '1  he  most  coirerv.itivc  set  is  the  set  that  has  the  lowest 
exjiected  Conti  ihtition.  I.C  . 


(6. 4. 2. 9) 


(sample  si^c 
On 

M = 1.365 
5 =.132 


of  15) 

Off 

M = 1.878 
5 = .277 


(the  expected  loo-ratio  is  1.25). 


Expected  Log-likelihood  Ratios 
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What  IS  the  expected  gain  from  gathering  another  fifteen  samples  from  the  operator’s 
distributions?  The  intervals  are 

(6.4.2.10)  (siimplc  size  of  30) 

On  Off 

1.258<:  M Si. 342  l.(/04S  M Si. 996 

.185s  S S.231  .203S  S S.254 

and  the  most  conseivative  distribution  (within  the  sixty  percent  intervals)  is 

(6.4.2.11)  (sample  size  of  30) 

On  Off 

M = 1 .342  M = 1 .904 

S =.185  S = .254 

(the  expected  log-ratio  is  1.80). 

1 he  potential  gain  is  significant  in  terms  of  the  increase  in  the  expected  log-ratio  for  the 
conservative  set  of  distributions  More  samples  would  increase  the  expected  log-ratio  even 
fuither  T(ie  upper  limit  on  this  log-ratio  would  be  reached  when  the  conservative  set  of 
distributions  was  the  same  as  the  sample  set.  At  that  point  the  expected  log-ratio  for  both  of 
them  would  be  ?.'ll.  The  number  of  samjiles  actually  used  in  a VV  task  depends  ujjon  how 
conseivative  the  programmer  is.  how  important  execution  time  is,  and  how  much  time  can  be 
devoted  to  training  the  system  Sample  sizes  on  the  order  of  twenty  to  fifty  have  worked  well. 

In  piogrammable  assembly  since  each  VV  task  is  performed  repeatedly,  it  is  possible  to 
gathei  additional  samples  during  production  runs.  This  is  important  because  a larger  set  of 
sam|jles  can  help  to  icfine  the  model  for  an  operator  in  two  ways.  First,  more  samples  can 
improve  the  distributions  being  used  to  model  the  operator,  and  second,  if  one  of  the  global 
variables  (eg.,  lighting  or  camera  sensitivity)  changes  slowly  over  time,  continuous  sampling 
can  maintain  an  up-to-date  model  for  the  operator. 
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C.t)\'C:i  USIONS  AN’n  1 XTF.N'SION’S 


'I'licic  aic  two  main  conclusions  ol  tins  thesis: 

(1)  A l.n^'i'  class  of  visual  leedUack  tasks  can  be  formulated  and 
aceomijlished  within  one  framework. 

(2)  I he  fiamewoik  can  be  im|>lementcd  in  such  a way  that  it  is  relatively 
easy  lor  a |>rcis'rammcr  who  is  not  an  expert  m vision  research  to 
cunstiuct  proeiams  that  pci  form  visual  feedback  tasks. 

1 fie  justificaiionsfor  both  of  these  statements  v.ere  essentially  proofs  by  construction.  A class 
of  visual  fcedl'ark  tasks,  icferrcd  to  as  vei ification  vision  tasks,  was  characterized  and  an 
inlet  active  system  ih.at  pieatiy  simplifies  the  (irOi^.ramming  of  such  tasks  was  imjjlemcnted. 


Section  I 

IRAMIWOHK  FOR  VV 


A VV  task  IS  a task  in  which  ilie  scone  is  highly  predictable,  there  are  no  big  sui|)iises 
'I  lic  laigest  single  step  involved  in  establishing  a fiainework  for  VV  is  the  development  of  a 
set  of  combination  rules  that  u.se  this  (iredictability  to  accomplish  the  task  m as  efficient  way 
as  possildc  Since  seveial  types  of  visual  operatois  have  been  used  extensively  in  the  past, 
tbeir  bcbaviois  arc  reasonably  well  understood.  It  is  also  relatively  clear  that  the  quantities  of 
Intel est  foi  ins|<cciion  and  location  tasks  are  confidences  and  precisions,  liut  it  is  not  clear 
how  to  combine  the  icsiilts  of  diffcient  operatois  in  oidcr  to  estimate  the  quantities  of 
interest. 


1 wo  sets  of  combination  rules  wcie  derived  iii  the  thesis  one  for  inspection  tasks  and 
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one  for  location  tasks.  Each  ot  these  sets  will  be  discussed  in  a separate  subsection  below. 

/.  Inspection 

The  inspection  rules  had  to  deal  with  lour  types  of  complications: 

(1)  Dillcient  operators  return  ditfeient  distributions  of  values  that  describe 
their  best  local  match.  One  type  of  operator  computes  correlation  values; 
another  computes  the  distinctness  of  an  edge. 

(2)  An  ojrerator  may  not  locate  a unique  feature  It  may  locate  any  one  of 
two  or  three  possible  features. 

(3)  An  operator  may  be  so  unpredictable  that  it  occasionally  finds  a 
completely  unexpected  feature  as  its  best  match. 

(■i)  An  operator  may  compute  seveial  values  that  describe  the  matching 
feature 

1 he  sti aightforwarcl  extensions  of  the  rules  to  cover  these  possibilities  involved  increasingly 
complex  formulas.  A set  of  assum|)tions  were  made  in  order  to  make  the  formulas 
computable  The  most  basic  assumption  was  that  traming  examples  were  available  from 
which  the  behavior  of  the  operators  could  be  approximated.  Fortunately,  training  examples 
are  available  in  several  types  of  visual  feedback  tasks,  eg.,  programmable  assembly,  satellite 
monitoring  of  crops,  and  chest  x-rays 

1 he  second  type  of  assumption  used  within  the  derivations  of  the  inspection  rules  is  the 
conditional  inde|)endence  of  some  of  the  information  produced  by  the  operators.  These 
assumptions  were  critical  because  they  reduced  the  intei dependencies  sufficiently  so  that  the 
rules  were  computable.  The  two  conditional  independence  assumptions  were: 

(1)  The  value  of  an  operator  is  conditionally  independent  of  the  values  of 
all  the  other  o|)eratois. 

(2)  The  value  of  an  operator  is  conditionally  independent  of  its  matching 
position 

Within  progiammahle  assembly  these  assumptions  were  found  to  be  approximately  true  for 
the  set  of  tasks  that  were  piog;ramnied.  The  first  assumption  is  often  true  because  different 
operators,  cs|iecially  different  types  of  operators,  depend  upon  different  properties  of  the 
pictures  The  second  is  generally  true  because  the  size  of  the  location  uncertainties  is  so  small 
within  VV  that  the  appearances  of  the  features  do  not  change  significantly  from  one  position 
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to  the  iii'M.  It  is  als.i  iniportant  to  note  that  thcip  aie  techniques  that  can  be  used  to  check 
the  validity  ol  these  assumptions  for  a specific  set  of  operators. 

P I ociinoK 

'I  he  comlunaiion  iiiles  foi  location  tasks  systematize  a collection  of  special-|jurpose 
lilies  that  use  the  loeaiion  of  one  or  more  fcitures  to  help  locate  other  features  Given  the 
location  of  one  hole  in  a subassembly,  the  system  can  predict  the  location  of  a second  hole.  In 
fact,  the  system  can  piucluce  a ranee  of  possible  locations  for  the  second  hole.  This  range  of 
locations  can  be  inteipieted  as  a precision  about  that  hole.  Precisions  are  important  for  two 
leasons  (1)  they  specify  infoimation  that  is  of  direct  interest  to  the  programmer  and  (2)  they 
can  be  used  to  cull  matclies  that  aie  inconsistent.  Culling  is  important,  of  course,  because  the 
opaerators  ate  not  compiletely  reliable. 

,A  IcMSt-squaies  technique  was  used  to  combine  the  position  information  for  a large 
numbet  of  matches.  1 here  arc  other  iiossiblc  metrics,  but  least-squares  is  a well  known 
technique  and  can  be  easily  extended  to  include  different  types  of  position  information,  such 
as  a point  on  a ciiclc 

1 he  location  lules  can  be  extended  in  at  least  two  ways:  (I)  they  could  incorporate  a 
wider  range  of  |.iosition  infoimation  and  (2)  tliey  could  adjust  a larger  set  of  parameters.  The 
cuiicnt  routines  only  piovide  for  pnint-to-pomt  matches.  Other  possibilities  include 
point-on-a-line  matches  and  point-on-a-circle  matches.  The  parameters  could  include  tire 
six  [laramciers  that  describe  a relative  change  in  the  location  of  the  object.  The  parameters 
could  also  rcjiresent  a relative  change  in  the  location  of  the  camera. 

The  location  rules  also  provide  a v/ay  to  check  the  consistency  of  a small  number  of 
matches  'I  he  consistency  is  based  upon  the  likelihood  that  the  object  would  be  at  the 
locatior^^plied  by  the  matches  If  it  is  impossible  or  very  unlikely  that  the  object  is  at  the 
imfrj^^Tlocation.  one  or  more  of  the  matches  must  be  incorrect 


Seel  ion  2 

FA.SF  OF  PROGUAMMINC 

1 1 he  difficulty  in  using  the  VV  system  cle|)end$  upon  two  factors;  (1)  the  number  of 

i stefjs  that  arc  not  completely  automated  and  (2)  the  quality  of  the  human  engineering  for 

j those  steps.  I he  semantic  structures  necessary  to  automate  some  of  the  steps,  such  as  ranking 
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opeiator/fcalure  pairs,  have  been  picscnted  in  this  thesis.  These  techniques  need  to  be 
improved  and  others  have  to  be  added  in  order  to  complete  the  automation  of  all  four  stages 
witliin  VV 

In  the  programming  stage  the  current  system  helps  the  user 

(1)  choose  potential  opcratoi/feature  pairs  by  applying  an 
interest  operator  to  the  picture, 

(2)  deieimme  the  correct  position  of  the  camera  by  computing 
the  task  toleiance  volume  and  by  predicting  the  expected 
precision  produced  by  the  operators, 

and  (?)  recalibrate  the  camera  each  time  a new  location  is  tried 

The  usri  has  to  state  the  task,  filler  out  suggested  operator/feature  pairs  that  the  system  does 
not  know  enough  to  avoid,  suggest  additional  operator/feature  pairs,  measure  the 
three-dimensional  position  of  the  features  on  the  objects,  and  manually  adjust  the  location  of 
the  camera 

1 he  system  can  be  easily  extended  to  adjust  the  location  of  the  camera  automatically.  A 
pair  of  cameras  and  the  associated  stereo  techniques  can  be  used  to  measure  the 
three-dimensional  positions  of  the  features  Rut  it  is  considerably  more  difficult  to  improve 
the  opei ator/feature  pair  suggestion  subsystem  Two  types  of  suggestion  techniques  were 
discussed  (1)  apply  interest  operators  to  typical  pictures  in  order  to  locate  visually  distinct 
fe.atures  and  (2)  use  models  of  the  objects  to  predict  visually  distinct  features  The  analysis  of 
pictures  needs  to  be  extended  so  that  it  loc.ates  other  types  of  features  besides  correlation 
features  Curve  linking  and  region  growing  should  be  used  to  suggest  line  features,  curve 
fp.aiures,  and  region  featuies  The  small  step  made  toward  model-based  suggestions  should  be 
completed  and  extended  to  produce  several  different  types  of  suggestions. 

At  tiainiiig  time  the  user  has  to  position  the  objects  in  the  scene,  take  a picture,  decide 
how  typical  the  arrangement  is,  and  indicate  whether  or  not  each  operator  locates  a known 
alternative  oi  a surprise  Some  of  the  object  jiositioning  can  be  automated  by  programiiiing 
the  mechanical  aims  to  sequence  through  a set  of  typical  locations.  The  structural  consistency 
tests  can  be  used  to  convert  the  reliability  decisions  into  a monitoring  operation  That  is, 
after  all  of  the  operators  have  been  applied,  the  best  structural  match  is  formed  and  the 
system  shows  it  to  the  user,  who  only  has  to  decide  whether  or  not  the  whole  stiucture  is 
correct 

1 he  current  planning  stage  ranks  the  operator/feature  pairs  for  inspection  tasks 
according  to  their  expected  contribution  (le,  their  expected  log-likelihood  ratio)  and  the 
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o|ici ;ttoi/lc.itiiio  p.iii.v  Id!  location  t.icl.s  accoiding  to  theit  telativc  structural  consistency. 
'I  lii-se  rnitliOi.ls  ate  usclul.  but  they  cio  i.ot  lake  several  iin(jortant  factois  into  account.  For 
c\ani|ilc,  they  do  not  consider  the  [lossibility  of  eNpancling  the  local  context  about  a match 
1 hat  IS,  It  the  lesults  of  ati  0|ici.itur  do  not  uniquely  determine  a match,  incorporate  the 
|iosition  intoiination  of  othei  o|ieiaiois  to  distinguish  between  the  alternatives.  A better 
strategy  subsystem  is  needed  that  develops  more  complete  plans  based  upon  better  estimates 
foi  the  expected  cost',  mme  flexible  VV'  piogtam  structures,  and  better  models  of  the 
tiadc-ofis  between  local  and  global  checking. 


' At  pro.  iamming  time  the  user  only  nionitors  the  progress  of  the  task.  However,  the 
system  can  still  be  impiovetl  so  that  it  provides  rnoie  complete  diagnostic  information  if 
something  tails  and  manual  intervention  is  requiicd.  For  example,  it  would  be  useful  if  the 
system  could  till  the  usei  wihat  it  expects  the  cm  rent  state  of  the  world  is  and  why  an  error 
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APPfNDIX  I.  ASSIGNMENTS  FOR  TWO  OPERATORS 


1 Ins  a|i|ieiiclix  develops  the  foiniula  for  the  probability  that  an  assignment  of  known 
alternatives  to  the  results  of  two  operatois  is  correct,  given  the  position  and  value  information 
produced  by  the  opeiators.  Assume  that  operator  1 Iras  M known  alternatives  and  operator  2 
has  N known  alici natives.  Let  fl,f2,...fM  be  the  alternatives  for  operator  1 and 
gl  ,g2,  . . .(jN  be  the  alternatives  for  operator  2.  Then  Bayes’  theorem  states: 

1 

(I.l)  I’lfj.gk  1 vl,pl.v2,p2]  = . 

P[vl,v2.pl.p2h(fj.gk)]  P[-(fj.gk)] 

1 + * 

P[vl,v2,pl,p2|fj,gk]  P[fj,gk] 

T he  piobability  P(  fj,gk|vl,pl,v2,p2]  represents  the  probability  that  the  first  operator  has 
located  alternative  fj  and  that  the  second  operator  has  located  alternative  gk,  given  the  value 
and  position  infomiation  produced  by  the  two  operators.  In  other  words,  it  is  the  probability 
that  the  assignment  of  f j to  operator  1 and  gk  to  operator  2 is  correct. 

The  standaid  assumption  about  the  conditional  independence  of  the  value  and  position 
information  can  be  used  to  reduce  (I  I)  to 

{1.2)  Pffj.gk  I vl.pl.v2.p2]  = 

. 1 


P[vl.v2|-,(fj,gk)]  P[pl,p2|-.(fj,gk)]  P[-(fj.gk)] 

1 4.  i, * 

P[vl,v2|f j.gk]  P[pl,p2|f j,gk]  P[fj,gk] 


or 


I 


I 
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' (1.3)  P[fj.gk  I vl,pl,v2.p2]  = 

1 


P[vl|-(fj.gk)]  P[v2h(fj.gk)]  P[pl.p2.-(f  j,gk)  ] 


P[vl|fj]  P[v21fj]  P[pl,p2,fj.gk] 

The  probability  P[-«(f  j.gk)]  can  be  expressed  as 

(1.4)  P[-(fj,gk)]  = y P[fx.gy]. 

-’(x=j  & y=k) 

That  is.  the  piobability  that  at  least  one  of  the  assignments  is  incorrect  can  be  expressed  as 
the  Slim  of  the  probabilities  of  all  of  the  other  possible  assignments  besides  [fj.gxj  Given 
this  expression  for  P[  -’f  f j , gk ) ] it  is  possible  to  express  P[pl,p2,->(fj,gk)]  as 

(1.5)  P[pl.p?,-(rj,gk)]  = y Prpl.p2,fx,gy] 

-’(x=j  & y=k) 

(see  section  3.5).  T he  probability  P[  vl  | -.( f j , gk)  ] can  be  expanded  in  a similar  way.  Since 
the  value  of  vl  is  assumed  to  be  independent  of  gy,  some  of  the  conditional  probabilities  can 
be  simplified. 


P[vl,  -.(fj.gk)] 

(1.6)  P[vl  1 -^(fj.gk)]  = 

P[ -'(fj.gk)] 


or 

y P[vl,fx,gy] 

-'(x=j  & yrx) 

(1.7)  P[vl  I -(fj.gk)]  = 

P[-(fj.gk)] 

or 

j y (P[vl|fx]*P[fx,gy]) 

-(xrj  & y=k) 


(1.8)  P[vl  I -(fj.gk)]  * 

P[-’(fj.gk)] 


'4 


Pfli.r  17-i 


Smct’ 

i 

fiy,  -.(fj.gk)] 

(I.O)  n[  fx.gy  I ^(fj.gt.)]  = . 

(-[-(rj.gk)] 

and 

(1.10)  ('[fx,  gy,  -.(fj.gk)]  = i[fx.  gy]  (if  -.(x=j  & y=k)), 
tciiniula  1 g can  be  rewi  iaen  as 

r'[fx,  gy] 

(1.11)  nLfx.gy  I -.(fj.gk)]  = (if  -(x=j  & y=k)). 

P[-(fj.gk)] 

1 ticn  tuiiiuil;i  18  sini(ilini's  to 

(1.12)  I’l'/l  I -.(fj.gk)]  = (l'i;vl|fx]*l'[fx,gy|-.(fj,gk)]). 

-.(x-j  8 y-k) 

Koiniulns  I 8 and  i Ifl  c.in  be  need  to  rcwiite  foiniiila  171  as 

(1.13)  l’[fj.gk  I vl  ,pl  ,v2,|)2]  = 

1 


V(r'[vl|ft]*0[r,s])  * V(PL  v2|(|x]*0[w.x])  * ^ P[  pl , p2 , fy , git  ] 

-.(r  = J & s=k)  -.(w=j  & x=k)  -.(y=j  & 2=k) 

1 ♦ 

I’lvllfj]*r'[v2|gk]*l'[pl,p2.fj.gk] 


wbfn-  01  r , s J I.  |ii(  Sfiits  P(  fr  ,gs|  -.(  f j,gk)  ].  roimiila  113  can  be  further  sir:i|jlified  to 


I 
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(I.H)  P[fj,gk  I vl,pl.v2,p?]  = 

1 


P[vl|fr]  P[v2|gx]  P[pl ,p2, fy.g/] 

'S  ^ ^ *0[r,s]* *0[w,x]* 

1 -.(r  = j -'(w=j  -.(y=j  F[vl|fj]  P[v2|gk]  P[  pl , p2 , f J , gk  ] 

& s=k)  & x=k)  & z=k) 

which  is  convenient  because  the  in-iportant  probabilities  form  ratios  that  can  be  easily 
evaluated  from  the  density  functions. 


I'av;c  I'/C) 


II 


ATPi-NDIX  II  r.lsT  KNOWN  AL1  LRNATIVHS 


'l  liis  ;i|i|ifiiiiix  cU'vcIo|.is  a fuii.iiila  ih.t!  iiicui |ioi .ites  ati  Ojjeiator's  position  information 
into  tlic  mechainsm  that  decides  wlnrli  I.novn  alti-, iiadvc  is  the  best  match.  The  previous 
method  made  this  decision  solely  uj.on  the  opti.iioi's  local  value  information  (sec  formula 
'1  hat  is,  tlie  altei  native,  fj,  with  the  hijlifsi  valiii'  of 

(11.1)  f'lrj  I v] 

W'as  chosen  as  tlie  best  maich. 

Considi’i  (he  fol/iiwinf  probability; 

(1 1. 2)  Pffj  I v.p] 

vi'fieie  p lepre'^ent'.  the  position  of  the  match  for  the  ojiciator,  the  fj's  ( j>0)  represent  the 
known  alternatives  foi  the  Ojreraior,  and  fO  rci'i  i sents  the  sui  prise  (if  any)  for  the  operator. 
Bayes’  theorem  states: 

1 

(11. 3)  l-lfjlv.p]  . 

I'lv,p|-.fj]  * 

] + 

I’lv.plfj]  * f'[fj]  . 


which  can  be  fx|ianciii.l  into 

1 

(II. 4)  PIfj|v.p]  = . 

Plvhfjtp]  * I'[p|-.rj]  * p[-fj] 

1 

('[v|fj;p]  * P[p|fj)  * KfJ] 

Usin^’.  the  standaid  assum|ition  that  thr  v's  aie  coiiUiiionally  independent  of  the  p's  this 
becomes 


f 


LJWL5J. 


♦ 


11 
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1 


(II. 5) 

P[fj|v,p]  = 

P[vhfj]  * P[p,-.fj] 

1 + 

P[v|fj]  * P[p,fJ] 

Since 

(II. 6) 

P[-fj]  = ^ P[fk]. 

then 

(II. 7) 

P[p.-fj]  = Z PtP.fk]. 
kxj 

Since 

(II. 8) 

P[v.-.fj] 

P[v|.^fj]  = , 

P[-fj] 

the  probability  P[v|-.fj]  can  be  expanded  in  a way  similar  to  (II. 7).  Then  formula  II  5 can 
be  rewritten  as 

1 

(11.8)  P[fJ|v.p]  = . 

P[v|fk]*P[fk] 

y * ^ P[p.fk] 

k#j  P[->fj]  k#j 

1 ♦ 

P[v|fJ]  * P[p.fJ] 

which  reduces  to 


(II. 9) 


P[f Jlv.p] 


1 


P[v|fin]  P[p,fn] 

1 . ^ ^ . 

mkj  nkj  Ptvifj]  P[p,fJ] 


P[fm] 

P[-fJ] 


i 
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All  of  tlio  coni|iC)ii('iit  |.iiol.ialjiliiics  in  tliis  foinuila  have  been  discussed  except  the 
P[p|fJ]  (fcM'  nil  j)  Whnt  should  their  vnliics  be?  One  possibility  is  to  use  tlie  original 
constraints  on  the  object  to  develop  an  a foiori  piobability  that  a known  alternative  will 
appear  at  a ceiiain  location.  Another  possibility,  if  sufficient  statistics  are  available,  is  to  use 
the  densities  obsei  veil  duiiiig  the  training  session  to  estimate  the  probability  of  finding  a 
match  at  that  point 


1 he  piohahihty  for  the  (i  e.,  P[plfO]'i  has  to  be  computed  separately  because 

theie  is  no  single  position  associated  with  a surprise.  It  can  be  appioximated  as  follows: 


< 

(II. 10) 


<llio  sub-re<iion  Unit  is  consistent  with  the  previous  featurcs> 
<tho  original  tolerance  region  about  fl> 


This  ratio  is  an  estimate  of  the  prob.ibility  that  the  surprise  will  accidently  be  consistent  with 
the  pievious  feaiuics  ’1  he  moic  features  that  liave  oeen  matched,  the  harder  it  is  for  a 
sur|ri  ise  to  look  consistent. 

The  lesult  of  this  apps'iidix  is  an  impioved  foimula  to  be  used  to  decide  which  known 
alternative  is  the  best  maicli.  T he  foirnula  is  an  improvement  because  it  is  based  upon  the 
position  information  of  an  ojjcr.ator  in  addition  to  the  value  information. 
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APPENDIX  III  EXPECTED  LOG  LIKELIHOOD  RATIO  FORMULA 


Coiiiick'i  an  Ojieratoi'  that  is  to  be  used  in  an  inspection  task.  If  there  are  no  known 
alternatives  or  snr|)riscs  for  the  operator,  and  if  the  operator  produces  values  that  are 
noinially  distributed,  what  is  the  CNpccted  value  of  the  logarithm  of  its  likelihood  ratio?  Tfiat 
is,  wfiat  IS  the  expected  contribution  of  the  operator?  Given  the  means.  Ml  & M?,  and 
srandaid  deviations,  SDl  &■  5D2,  for  the  two  distributions  and  the  a priori  probability  that  the 
object  IS  present,  P.  there  is  a simple  formula  that  computes  the  desired  quantity. 

This  section  derives  the  appropriate  foimula.  After  some  initial  simplifications, 
MACSYMA  is  used  to  perform  the  necessary  symbolic  integration. 

T he  expected  value  can  then  be  computed  as  follows: 


+ CO 

(Ill.l)  expect€d_log-ratio  s J log-ratiofX)  * denslty(X)  dX, 

-CO 


where 


(III. 2) 


(III. 3) 


On_density(X) 

log-ratio(X)  = log  ( ), 

Off_dcnsity(X) 

density(X)  = P*On_dcnsity(X)  + ( 1-P)»0f f_densi ty( X) , 


On_dcnsity(X) 


2 

1 (X  - Ml) 


2 2 
SOI 
* e 


(III. 4) 


S01»S0RT(2*ti) 


Pa£;e  ISO 


1 (X  - M2) 


2 2 


(III. 5) 


Off_c)cnsity(  X)  = 


S02nSORT(2<.n) 


Foi  nuila  1112  can  bo  siin|.ilitit(.l  as  follows: 


1 (X  - H2)  1 (X  - Ml) 

(-S - -* ) 

2 2 2 2 

.S02  SD2  5D1 


(III. 6)  log-ratio(X)  = logf  *■  o 


1 (X  - M2)  1 (X  - Ml) 

(III. 7)  lo(j-ratio(X)  = loo(5D2)  - loa(5Dl)  + -* - -* 

2 2 2 2 


Since  the  iniej'.ial  fioni  minus  infinity  to  plus  infinity  of  a density  function  is  1.0,  (111  1) 
reduces  to. 


(III.O)  cxpcctod_log-rnlio(X)  = log(SLi2)  - log(SDl)  + 


■foo  1 (X-M2)  1 (X-Ml) 

f ( — * - — * ) * ( l’*0n_dpnsi  ly(  X)+(  1 -P)*0ff_densi  ty(  X)  )dX . 

-«  2 2 2 2 


The  integral  term  can  be  factored  into  four  smaller  integrals  that  arc  all  essentially  the  sanre 
Their  general  foi m is: 


111 
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1 (X  - M) 

— * 

2 2 


(HI. 9) 


+(»  1 (X  - H) 

1(5, M)  = f — * * 


-00  2 


S*S0RT(2*n) 


* e 


clX. 


MACSYMA  was  used  to  conijjute  I(S,M)  and  the  final  formula.  A trace  of  the  session  with 
MACSYMA  is  presented  below: 


THIS  IS  MACSYMA  259 

FIX  259  D5K  MACSYM  BEING  LOADED 
LOADING  DONE 

(Cl)  f (X.S,M):  = (l/(S*S0RT(2*y.PI)))*7.f  i(-l/2*((X-M)/5)f2); 


(01) 


X - M 2 

- 1/2  ( ) 

1 S 

F(X.  S,  M)  ;=  XE 

S S0RT(2  XPI) 


(C2)  G(X,S.M):=(1/2*((X-M)/S)t2): 


(02) 


1 X - M 2 

C(X,  5.  M)  :=  - ( ) 

2 S 


(C3)  P*G(X,S2,M2)*F(X.S1.M1): 


Page  \S2 


(D3) 


2 

(X  - Kl) 


2 

2 ? 51 

P (X  - M^)  %i 


2 

2 SQRT(?)  SO^^T('XPI)  51  52 


(C4)  lNltGRA'IL(D3.X,MlNr.lNr  ) ; 


DEFINT  FASL  DSK  MACSYM  BEIIJG  LOADED 
LOADING  DONE 

LIMIT  FASL  DSK  MACSYM  BLINC  lOADEO 
LOADING  DONE 

< 

RE5IDU  FASL  DSK  MACSYM  BEING  LOADED 

LOADING  DONE 

Is  51  zero  or  nonzero? 

NONZERO: 


3 

(04)  P (2  SQRT(2)  50RT(y.PI)  51  + 50RT(2)  50RT(y.PI) 


2 2 2 
(2  M2  - 4 Ml  M2  + 2 Ml  ) Sl)/(4  SQRT(2)  SQRTCXPI)  SI  S2  ) 

1 


(C5)  RATSIMP(X): 


Ill 


Page  IS? 


2 2 2 
P SI  + (M2  - 2 Ml  M2  + Mi  ) P 

(D5)  

2 

2 S2 

(C6)  FACTOR(%): 

2 2 2 
P (SI  + M2  - 2 Ml  M2  + Ml  ) 

(D6)  

2 

2 S2 

(C7)  R(SD l,MNl.SD2,MN2):=(SD2t2+(MNl-MN2)t2)/( 2*501 t2 ) : 

2 2 
SD2  ♦ (MNl  - MN2) 

(07)  R(SD1,  MNl,  S02,  MN2)  

2 

2 SOI 

(C8)  P*R(S2,M2,S1,M1  ) + (l-P)*R(S2.M2,S2.M2)- 

P*R(S1.M1.S1,M1)-(1-P)*R(S1,M1,S2,M2): 

2 2 2 2 
(1  - P)  (S2  + (Ml  - M2)  ) P (SI  ♦ (M2  - Ml)  ) P 1 - P 

(08)  ♦ 

2 2 2 2 
2 SI  2 S2 

The  final  formula  for  the  expected  contribution  of  an  operator  that  produces  normally 
disti ibuted  values  is: 


Pape  IS-j 


(III. 10)  cxpcclcd_1og-rat1o  = lo(|(Sn2)  - lOfj(SDl)  + 1/2  - P 


2 2 2 2 
SDl  + (M2  - Ml)  SD2  + (H2  - Ml) 

+ I'* — - (l-P)* 


2 * .sn2 


2 * SOI 


Page  IS') 


APPENDIX  IV.  SCREW-INSPECTION  EXAMPLE 


Appendices  IV  and  V are  designed  to  demonstrate  the  capabilities  of  the  current 
implciTientation  of  the  VV  system.  Both  of  the  appendices  consist  of  annotated  traces  of  the 
Intel  action  between  a programmer  and  the  VV  system.  Appendix  IV  describes  the  standard 
screw-inspection  task.  Appendix  V describes  the  location  of  the  screw  hole. 

Appendix  IV  has  four  sections: 

(1)  The  fust  section  is  a trace  of  all  four  stages  for  the  screw-inspection  task: 
firogrammmg,  training,  planning,  and  execution. 

(2)  At  the  end  of  the  planning  stage  the  program  outputs  a file  that  contains 
all  of  the  information  needed  to  perform  the  task.  The  second  section 
discusses  this  task  file. 

(3)  In  the  first  section  a calibration  object  is  used  to  calibrate  the  camera 
However,  since  the  calibration  is  essentially  another  VV  task,  it  is 
possible  to  use  a location-type  VV  task  to  calibrate  the  camera  directly 
with  ies|ject  to  some  of  the  fixed  objects  in  the  scene,  such  as  the  screw 
dispenser  The  thud  section  shows  how  the  system  can  be  used  to 
perform  this  type  of  calibration  for  the  screw-inspection  task. 

(“1)  The  VV  program  that  performs  the  screw-inspection  task  has  been 
inteifaced  to  the  arm  contiol  system.  AL,  The  fourth  section  describes 
the  interface  and  presents  the  arm  program  that  uses  visual  feedback  to 
check  for  a sciew  on  the  end  of  the  screwdriver. 

The  notafional  conventions  for  the  traces  are:  (1)  the  output  of  the  coiTipuii  i 
normally  left-justified,  (2)  when  a response  is  requested  fiom  the  user,  the  request  ends  in 
colon  and  the  user's  response  immediately  follows  the  colon,  and  (3)  the  comments  tha:  ha 
been  added  to  the  trace  are  in  italics  and  are  indented  seveial  spaces 
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Scclioii  1 

TKACL  OF  A SCUnW’-JNSI'FCTlON  TASK 


■|  his  tr^trr  dcmnnstt arcs  tin'  inti  isciiiin  b<'twfcii  a programmer  and  the  VV  system.  Ot 
particular  interest  is  the  planning  stage  that  ranl.s  the  potential  operator/feature  pairs  and 
estimates  the  e.'s|ieitt'd  numher  of  o|ii  laiuts  rcipiifil  tu  achieve  a certain  cotifidence 

I'itOtiltOh'ilNf.  Ilttr 


1 hp.  luoqroi’ininq  si.a<ie  iin'oli/rx  (1)  calibrating  the  camera  to 
the  I'.iirkMat  io/i  ainJ  (?)  poinLing  out  distinctive  rcaturcs  on 
the  object  of  interest.  Operutors  ivill  be  used  to  try  to 
locate  these  rcaturcs.  I he  results  of  the  operators  v/ill  be 
used  to  decide  whether  or  not  the  object  is  present. 

Do  yon  want  tn  use  the  cnninra?  (V  or  H):tl 

A ye.s  answer  to  this  question  neons  that  live  pictures  from  the 
camera  will  he  taken  when  they  are  needed.  A no  answer  means 
that  stored  disk  pictures  will  he  used. 

Ca  1 1 hr  tat  ion  p ' r.t  me  t.SCAl.  1 . 1’J  C 

SCAll.Pir  is  a pictwr.  of  the  calibration  ohjcct,  which  tias 
hrin  placed  at  a knov.i,  location  within  the  workstation  (sec 
f iqiire  Id  .1 . 1 ) . 

Display  on  the  syn  t he.s  i7r  r , halftones,  or  none?  (S,ll,or  N):H 

IHL  I’lCTUKl. 'S  ( llAIJIll  1 AIJD  lt!|  OV!  Kl  AY  OiAMtl  I : 2!  28 

I hr.  .synl  lies  i/r.r  and  halt  ton.-,  displays  are  two  dirrcrenl  ways  to 
display  the  pictuics  of  interest.  both  methods  display  two 
pictures  s ide-by-s  idc. . llic  planning  picture  is  normally  on  the 
left  and  the  test  picture  is  on  the  right. 

Want  to  use  a calibration  file?  (Y  or  N):fl 

Since  the  calibration  file  is  about  to  be  defined,  this 
question  is  answered  with  a no. 

Start  calibrating  the  car.cra. 

Sire  of  the  iiiterc.st  operator  ( A tO 

1 hr.  interest  operator  is  the  operator  that  is  applied  to  a 
* planning  picture  in  order  to  locate  promising  features. 

Size  of  the  correlation  operator  (on.  ti)ih 

Automatic,  Mtiniial,  or  .Standard  Cal  ibration?  (A,  M,  or  .S):M 

I he  standard  cal  ihratioii  is  the  cal  ihration  for  the  camera  v/hen 
it  is  aimed  at  the  middle  nf  the  workstation.  It  is  used  when 
the  precision  of  the  cal ihration  is  not  important . 

Ihc  aiiLonatic  calibration  uses  a calibration  file  to 
recalibrate  the  camera,  which  is  assumed  to  be  close  to  the 
location  specified  in  the  file.  Ihc  default  calibration  file 
is  the  standard  cal  ihration  file. 

Ihc  m.uiual  calihiatwn  asks  the.  user  to  point  out  features  in 
the  picture  of  the  calibration  object.  Ihc  interest  operator 
IS  applied  tn  tiu:  picture  of  the  calibration  object,  one 
.suqqrstion  at  a time  ir,  shown  on  the  display,  and  the  user 
( boost  s thr  featuies  In.  likes.  Ihe  programmer  has  se.vcral 
111 tvrnal  ives  when  ore  sen  Led  with  a suggestion ; (1)  accept  it, 

(?)  n jecL  it,  (.1)  locally  adjust  its  position,  (4)  apply  the 
associated  operator  to  the  test  picture,  on  the  right  (if  there 
IS  one),  (b)  display  its  screen  and  workstation  coordinates, 
(h)  apply  the  associated  op  ralor  to  the  trial  picture  on  the 
right  in  order  to  find  the  three  or  four  host  alternatives  (in 
addition  to  the  expected  match),  and  (/)  stop  considering 
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Figure  IV. 1.1 
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smuiiwt  ionf.  proOiiccJ  hy  the.  inLcrc.’it  operator.  the  Sana 
.•cell'd,  ion  firoccs.^  is  used  Lo  define  features  for  a task  as  for 
a calibration. 

the  system  displays  the  first  suodcstion  in  the  planning 
picture  and  presents  tnc  user  with  the  following  menu  of 
responses , 

Accept, R^rjecL.Loccilly  adjust, Try, Coorcliniitc  display. 

Find  best,  or  Stop  accepting  suoiicst ions  (A,R,L,T,C,F,  or  S):L 
X and  Y:-2 
X and  Y; 

1 he  user  moved  the  suggestion  two  pixels  to  the  left  in  order 
to  center  it  hitter  on  the.  feature  of  interest.  The  null 
response,  after  the  second  reiiucst  for  an  X and  Y is  simply  a 
carnage  return,  <cr>,  which  indicates  that  there  is  no  further 
adjusl.ificnt  desired. 

Accept , Reject , 1 nca  1 ly  iid just , 1 ry , Coord ir..; to  display. 

Find  best,  or  Stop  arceptin':i  su'iocst  ions  (A,R,L,1  ,C,F,  or  5):A 
The  feature's  X,Y,/;  .5  1.5  0 

the  user  specifies  the  varhstation  coordinates  for  each  feature 
that  hr  accepts.  Given  the.  screen  coordinates  and  the 
workstation  coordinates  for  several  features,  the  program  can 
calibrate  the  caiiera. 

Accept , Re  joct , I oca  1 ly  adjust , Iry,  Coord  mate  display. 

Find  best,  or  Stcip  arceptinu  siKidCSt  ions  ( A,R,  L,  1 ,C,F , or  S):A 
The  fCiiturc's  >:,Y,/:  .5  .5  0 

fitter  two  or  three  suggestions  the  system  ahhreviat  ^ menu. 
(A,R.L,l,C,i,  or  S ) : L 
X and  Y:-tO  -1 
X and  Y;+0  -P 
X and  Y: 

(A,R,L ,1 ,C,f , or  S);A 
Ihe  feature's  X,Y,/:  1.5  .5  0 
(A,R.L,1 ,C,I  , or  S):L 
X and  Y:-?  +2 
X and  Y: 

(A,R.L,1,C,F , or  S):A 
The  feature's  X,Y,/:  0 0 0 
(A,R,l ,T,C,F,  or  S):R 
U.R.L.T.C.I  , or  S):L 
X and  Y;+0  -/ 

X and  Y: 

(A.R.L,T.C,F,  or  S):A 
The  feature's  X,Y,Z:  2.0  2.0  0 
{A,R,L,T.C,F.  or  S):R 
(A,R.L.T,C,t  , or  .S):.S 

Do  you  want  to  point  out  any  additional  features?  (Y  or  N):Y 

Ihc  user  decided  to  stop  considering  the  suggestions  of  the 
interest  operator  and  to  specify  one  more  feature.  Currently 
typed  numbers  arc  u.sed  to  position  and  adjust  the  cursor.  A 
joystick  or  mouse  would  he  more  efficient . 

X and  Y for  new  feature  (in  pixels)  (<cr>  to  stop):40  75 
(A.R,L,T,C,F.  or  S):l 
X and  Y:+0  -1 
X and  Y: 

(A.R.l  ,T,C,I  . or  .S):A 

The  feature's  X,Y,/:  1.5  1.5  0 

X and  Y for  ntw  fe.itiire  (in  pixels)  (<tr>  to  stop): 

figure  JV.l.P  shows  the  features  that  were  defined  on  the 
calibration  oh.icel. 

Approximate  lens  center  (X,  Y, / ) : -22 . 5 -o.2o  l'i.75 

Ihe  approximate,  position  of  the  lens  center  is  used  in  the 
camera  eal ihral ion  routine.  Given  that  information,  the  system 
can  calihrale  the  camera  and  output  the  calibration  file,  which 
contains  enough  ini orimit ion  to  recalibrate  the  camera 
automatically  from  a picture  taken  at  a slightly  different 
location.  Ihc  system  docs  not  immediately  output  the 
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Crilihration  file  because  the  camera  may  have  to  he  adjusted  in 
order  to  accomplish  the  desired  task.  Thus,  the  user  states 
the  task  and  makes  sure  that  the  desired  results  can  he 
produced  with  the  camera  at  the  current  location  before 
outputing  the  calibration  file. 

State  the  task.  Specify  the  type  of  task  and  the  known 

constraints  on  the  object  of  interest. 

Type  OF  TASK?  (0,  location;  1,  in.<ipf!Cl ion ) : 1 
Si?c  of  the  interest  operator  (4,fl,...):8 
Size  of  the  correlation  operator  (erj.  9):9 

7/175  version  of  the  1/1/  system  only  has  one  type  of  operator: 
Horavec's  correlation  operator. 

Known  limits  on  X,  (-clx,+clx);'.3  +.3 
Known  limits  on  Y,  ( -dy  .-icly ) : - . 3 +.3 
Known  limits  on  Z,  ( -d/ , •< d;' ) : * -3  ♦•3 

Expected  precision  of  the  correlation  operator,  (eg.  1 pixel):l 

Ihe  expected  precision  of  an  operator  is  the  average  distance 
between  the  estimate  of  the  location  produced  hy  the  operator 
and  the  actual  location  of  the  feature.  Given  the  expected 
precision  of  an  operator  in  a two-dimensional  picture  and  a 
camera  calibration,  it  is  possible  to  determine  the  expected 
three-dimensional  precision  about  a point  in  the  workstation 
coordinate  system.  for  example,  as  stated  below,  given  the 
current  calibration,  an  error  of  one  pixel  can  produce  as  much 
as  an  error  of  .0260d8  inches  along  the  X-axis  when  the 
correlation  tries  to  locate  the  lower  left  corner  of  the 
calibration  object. 

Do  you  want  to  determine  the  picture  size?  (Y  or  N):N 

The  above  question  asks  the  user  if  he  wants  to  use  the 

expected  precision  of  the  operators  to  determine  (1)  the  si/e 

of  the  picture  required  to  insure  that  the  features  of  interest 
will  he  in  the  picture  and  (2)  the  resolution  of  the  picture 
required  to  achieve  the  desired  precision.  Since  the  user  can 
easily  sec  that  the  uncertainties  associated  with  the  screw 
will  not  move  the  screw  outside  the  current  picture,  and  since 
precision  is  not  important  in  this  case,  the  user  decides  not 
to  determine  the  picture  size. 

If  the  user  is  not  sure  about  the  size  and  resolution  of  the 
picture,  hr  can  use  the  system  to  determine  them.  The  camera 
may  have  to  he  moved  and/or  zoomed  so  that  It  takes  the  correct 
pictures.  If  it  is  adjusted  in  any  way,  it  has  to  be 
recalibrated.  Appendix  V presents  an  example  of  this 

adjustment  and  recalihration  process. 

Since  the  u.scr  decided  not  to  adjust  the  camera  for  this  task, 
the  system  states  the  current  expected  three-dimensional 
precision  of  a correlation  operator  centered  on  the  origin  of 
the  calibration  object  and  then  asks  the  user  if  he  wants  to 
save  the  calibration  file. 

X,Y,7  precision  of  op:  .02247?  .022626 

Do  you  want  to  save  this  calibration  file?  (Y  or  N):Y 
Output  file  name : IV/. CAL 

7/ic  calibration  file  has  been  completed.  If  the  camera  is 

moved  to  take  a picture  for  another  task  and  then  moved  hack  to 
approximately  the  same  location  for  the  screw-inspection  task, 
this  file  can  be  used  to  recalibrate  the  camera  automatically. 

Use  « task  f11c?.(Y  or  N):N 

Choose  the  potential  operator /feature  pairs  for  the  screw 
inspection . 

Planning  p1cture:S('LANl  .PIC 
Picture:5IESTl.PIC 

The  two  pictures  are  shown  in  figure  IV. 1.3.  The  first 
suggestion  produced  by  the  interest  operator  is  also  shown  on 
the  left. 

Accept, Reject, Local ly  adjust, Try, Coordinate  display. 

Find  best,  or  Stop  accepting  suggestions  (A,R,L,T,C,F,  or  S):T 
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CORRELATION  VAl  IJE  .92? 

[och  sii(j<icstion  dcTincs  an  operator /feature  pair.  The  feature 
is  the  part  of  the  object  that  produces  the  portion  of  the 
picture  covered  by  the  suggestion . The  operator  is  the 
correlation  operator  defined  by  the  array  of  intensity  values 
in  the  suggested  portion  of  the  picture. 

The  user  decided  to  test  the  operator  defined  by  the  first 
suggestion . Ihc  operator  was  applied  to  the  test  picture  and  a 
dot  was  placed  on  its  best  match  (see  figure  I\J.1.4).  The 

value  at  the  best  match  also  printed.  If  the  value  is  too 
low,  or  if  the  operator  finds  the  wrong  feature,  the  user  may 
decide  to  discard  the  feature,  locally  adjust  it,  or  try  to 

characterize  the  known  alternatives . 

Accept, Reject,  Local  ly  nclju.st, Try, Coordinate  display. 

Find  best,  or  Stop  accepting  suggest ion.s  (A,R,L,T,C,F,  or  S);C 

The  user  asks  for  the  coordinates  of  the  current  suggestion, 
lifter  asking  for  the  height  of  the  feature  the  system  states 
three  sets  of  coordinates  for  the  feature:  (a)  the  pixel 

position  within  the  picture,  (h)  the  position  within  the 
picture  stated  in  terms  of  percentages,  and  (c)  the 

three-dimensional  position  within  the  calibration  object's 
coordinate  system. 

Helaht  of  the  fcature:0.0 
PIXELS  (X,Y):  46  111 
(0  - 1 )(X,Y  ; .460  .74.3 

3-D  (X.Y,7);  .441  .967  .000 

Accept, Reject, Local ly  adjust, Try, Coordinate  display. 

Find  best,  or  Stop  accepting  suggestions  (A,R,L,T,C,F,  or  S):A 
Z:0.0 

Accept, Reject, Locally  adjust, Iry, Coordinate  display. 

Find  best,  or  Stop  accepting  suggestions  (A,R,L,T,C,F,  or  S):T 
CORRFIAIION  VAl  LIE  .890 

Accept, Reject, Locally  adjust, Try, Coordinate  display. 

Find  best,  or  Stop  accepting  suggestions  (A,R,L,T,C,F,  or  S):A 
2:2.5 

(A,R,L,T,C,F,  or  S):T 
CORRELATION  VALUE  .913 
(A,R,L,T,C,F,  or  S):L 
X and  Y:+i  -1 
X and  Y; 

(A,R,L.T,C,F,  or  S):T 
CORRELATION  VAl UE  .937 
(A,R,L,T,C,F,  or  S):A 
2:  ,5 

(A,R,L,T,C,F,  or  S):L 
X and  Y:-2 
X and  Y: 

^A,R,L,T,C,F,  or  S):A 

(A,'r,L,T,C,F,  or  S):L 
X and  Y:-4  -1 
X and  Y: 

(A,R,L,T,C,F,  or  S):A 
2:  .75 

(A,R,L,T,C,F,  or  S):L 
X and  Y:*0  -2 
X and  Y : 

(A,R,L,T,C,F,  or  S):T 
CORRELATION  VALUE  .934 
JAjR^L,T,C,F,  or  S):A 

Ihe  system  ran  out  of  good  suggestions,  so  It  asks  the  user  if 
he  would  like  to  define  any  other  features  interactively. 

Do  you  want  to  point  out  any  additional  features?  (Y  or  N):Y 
X and  Y for  new  feature  (In  pixels)  (<cr>  to  stop):40  60 
(A,R,L,T,C,F,  or  S):L 
X and  Y:*2  -5 


IJr: 
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X find  Y;  + 0 -1 
X n n d Y ; 

( A . R . I , 7 . r , r , or  S ) ; 7 
C0RrsF.LA710M  VAI  UE  .7»31 
(A,R.L,7,C.F,  Of  S):A 
Z:  .7b 

X find  Y for  now  fo.it.iirc  (in  pixels)  (<cr>  to  .stop): 

Approx if.i.itc  screen  coords  of  tlio  (>oint  of  inttTesL  (x,y):A5  110 
X <ind  Y:-il 
X and  Y: 

The  Z of  the  point  of  intern;, t,  7:0.0 
X,Y,Z  prcci.sion  of  op:  .070  .0?3  .02? 

I i (Hire  s)iiiv's  Lite  feniiirns  rir.rinnd  for  the  task.  Notice 

ili'iiL  one  or  the  fc.iiiircs  ( niirihcr  ?)  is  on  the  screwdriver . It 
IS  noi  expected  to  contr  Unite  very  much,  hut  it  is  included  to 
show  how  the  runking  procr:.;,  [ perl  orned  during  planning  time) 
rotes  such  fcoturrs.  Ir.ot.urr.  nuthcr  ti  is  similar.  It  is  hosed 
upon  the  screw  dispenser  nnd  h,.iice  should  not  be  able  to 
d 1st  inginsh  In  iweru  the  s i iuotioi;  in  which  the  screw  is  on  the 
end  of'  the  screwdr  iver  o:n!  the  situation  in  which  the  screv/  is 
irissino.  froturc  iniriin-.r  J is  hosed  upon  the  tip  of  the  scree.'. 
Since  the.  tip  of  the.  sri  ,.wdr  ivi.i  lonks  very  similar  to  the  tip 
of  the  screw,  opi.i  otnr  J is  not  expected  to  be  very 
(list  iiigti  ishing  either. 
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fiuriii'j  the  training  session  the  operators  are  applied  to 
scveitil  example  picLores  and  Ihn  results  arc  used  to  form  the 
expected  distr  ihutions  ol  volue.s  for  the  operators  and  the 
expected  rcl  lahil  itics  of  the  operators . 

Training  Pic  tor c :S I LSI  1 . PI C 

lor  each  training  picture  the  planning  positions  for  the 
features  arc  shown  on  the  left  and  the  actual  matches  arc  shown 
on  the  right  (:,ir.  I igure.  lU.l.O).  If  one  of  the  operators 
misses  its'  de.s  ignntcd'  fe.it  tire , the  user  specifies  its  number 
inid  the  progran  incorporate.:,  that  information  into  its 
rel  iahil  ity  slat  istics . llu:  ope.rators  did  not  miss  any  of 
their  features  in  this  treimog  picture. 

Do  you  v/ant  to  include  the  re.siiits  of  ttii.s  irifil?  (Y  or  N):Y 
fJunher  of  a fe.iturc  that  rii.s.serl,  (I!  or  <ci  >): 

Is  the  ohjcct  there  or  not?  (1  or  11)  :T 
Another  trial?  (Y  or  II)  :Y 
7 ra in  1 nu  P ic  tui o ; 5 1 1 S 12 . PIC 

Do  you  want  to  include  the  rer.nlts  ol  this  trial?  (Y  or  N):Y 
Nunher  of  a fratiiie  that  mi.s.sed,  (IJ  or  <rr'.');? 
lumber  of  a fe.Tture  that  ni.'.seil,  (N  or  <ri>): 

Operator  ? missed  its  di  s ignal i.J  feature  because  the  feature  is 
not  in  the  picture. 

Is  the  object  there  in-  pfit?  (I  or  IJ):1 
Another  trial''’  (Y  or  IJ ) ; Y 
Training  Pic  tore  :S  1 1 S 1 J . P If. 

Do  you  w.int  t'l  include  the  re;.iilts  of  this,  trial?  (Y  or  N):Y 
Humber  of  a foature  that  mi:..sej,  (II  or  <cr>); 

Is  the  object  there  or  not?  (I  or  li);l 
Another  trial?  (Y  or  f.’):V 
Training  P ic  lure  :S  I l.S  M . PIC 

figuro  JV.J.y  shows  SlLSld.PlC  in  which  the  screw  is  missing. 
« Ihc  overlay  shows  the  best  m.itch  for  each  of  the  operators . 

Operators  two  and  six  found  their  correct  matches.  The  others, 
since  they  are  hasr.d  upon  the  screw,  which  is  not  present,  can 
not  possibly  locate  their  correct  matches.  So  they  find 
some.lhing  else,  Ihesc  m.ikeshift  matches,  when  the  ohjcct  is 
missing,  arc  not  considered  to  he  mismatches  in  the  reliability 
statist  ICS  lor  the  opiraior.  A more  general  system  would 
assign  two  correct  feature  matches  for  each  operator:  one  when 


the.  object,  is  nfC-sent  end  one  when  the  object  is  missing.  fln 
even  more  general  system  would  allow  the  user  to  define 
features  in  two  planning  pictures:  one  with  the  object  present 
and  one  with  the  object  missing. 

Do  you  wnnt  to  Include  the  results  of  this  trial?  (Y  or  N):Y 
Number  of  a feature  that  missed,  (N  or  <cr>): 

Is  the  olijcct  there  or  not?  (T  or  N):N 
Another  trial?  (Y  ot  N):Y 
Training  Picture :S1  LSI t> . f'lC 

Do  you  want  to  include  the  results  of  this  trial?  (Y  or  N):Y 
Number  of  a feature  that  missed,  (M  or  <cr>): 

Is  the  object  tliere  or  not?  (T  or  N):N 
Another  trial?  (Y  or  N);Y 
Training  P1cturc;S1[S16.PlC 

Do  you  want  to  Include  the  results  of  this  trial?  (Y  or  N):Y 
Number  of  a feature  that  missed,  (fJ  or  <cr>l:2 
Number  of  a feature  that  missed,  (IJ  or  <cr>): 

Is  the  object  there  or  not?  (T  or  N);N 
Another  trial?  (Y  or  N):Y 


A total  of  forty  training  picture  were  used.  Given  the 
training  pictures,  the  training  only  takes  about  twenty  to 
thirty  seconds  elapsed  tine  per  picture. 


Another  trial 


op  # N 

1 19.000000 

2 15.000000 

3 20.000000 

4 20.000000 

5 20.000000 

6 19.000000 

7 20.000000 


? (Y  or  N);N 

Ihc  system  gathered  statistics  on  the  correlation  coefficient 
values  produced  by  the  operators.  It  actually  uses  the  change 
of  varitihle  discussed  in  section  3.6.2,  which  produces  a 
distribution  that  is  closer  to  a normal  distribution . The 
change  of  variable  also  modifies  the  range  of  values  from 
[-1 .0  ,-tl  .0]  for  the  correlation  coefficients  to  for  the. 

changed  values. 

FINAL  STATISTICS 


( there ) 
MFAN 
1.238000 
1 .224000 
2.053000 
1 .917000 
1 .870000 
1.245000 
1.947000 


SI)  N 

.3400000  20.0000000 

.1830000  14.0000000 

.2850000  20.0000000 

.3370000  20.0000000 

.2660000  20.0000000 

.4150000  20.0000000 

.2140000  20.0000000 


(not  there) 
MEAN 
1.196000 
1 . 190000 
1 .252000 
1 .223000 
1 .170000 
1.143000 
1.312000 


sn 

.4030000 
.1500000 
.3110000 
.3280000 
.3620000 
.2540000 
. 1490000 


I'lANNirir.  TIMF 

The  main  purpose  of  the  planning-time  processing  is  to  decide 
which  operator /f eaturc  pairs  are  the  best.  That  is,  determine 
which  operators  are  expected  to  contribute  the  most  toward  the 
decision.  for  an  inspection  task  the  contribution  of  an 
operator/f  eaturc  pair  is  measured  by  its  expected 
log-likelihood  ratio.  The  log-likelihood  ratio  is  a measure  of 
the  operator ' s ability  to  distinguish  betv/een  the  iwo 
possibil  it  ics:  (I)  the  screw  is  present  and  (2)  the  .screw  is 
missing.  Ihe  larger,  the  better. 

After  computing  the  expected  log-likelihood  ratios  for  all  of 
the  operator/feature  pairs,  the  planning  stage  forms  an  ordered 
list  of  operators  to  he  applied  to  a picture.  This  ordered 
list  of  operator/f cature  pairs  is  essentially  the  program  to  be 
used  to  decide  whether  or  not  a screw  is  present.  Ihe. 
operators  will  be  applied  one  at  a time  and  their  results  will 
he  sequentially  incorporated  into  the  overall  probability  that 
the  screw  is  present. 

Given  the  ordered  list  of  operator/f cature  pairs,  the 
planning-timc  program  can  use  the  theorem  discussed  in  section 
6.6  to  estimate  the  number  of  operators  required  to  reach  a 
certain  confidence. 


Phs;.-  PIO 


IV. 1 


WIUll  IS  llK' 
ItIL  OiUil  Kl  II 
7 

3 ? . 

A 1 .r.n3 

5 4.3''.l 

6 .3 '.'3 

1 .Oiw' 

2 .O'Ui 


.1  i-riori  |)rob  lli.it 
SI  I 01  I I AlUKLS  . . 


t.lir  ot'.ii'Cl  IS  iirc'scnl ; . 9 
/f  ■;iiu  t,la'  rxpoctccl  locj- 1 1 kcl  ihood  ratio 


I'.nlici:  thill  the  three 
lo\"rr  riitirnis  l.h.in  the 
In  hi  (1  >;.tiniiin;,huni. 
li  istr  ihi'l  inns  r.ssnc.  irl  rd 
iirr  pn  'ti'iil.i'd  h ! l-ln-r 


duhintir.  features  arc  (liven  considerably 
cilh!.r  ! c:ir  Iraturcs,  w/n'c/i  were  expected 
I mill  i:  iy.J.8  shows  the  pairs  oT 
wilh  nil  ol  the  operators.  The  graphs 
'•phi.  and  inp- 1 (i-hottom  in  the  order 


lin'd  hy  till 
nrs  piodllii 


I ' I . 
iipr  I'ii 

pi  e nl.  and  a 
(I  IS  t r ituii  ions 


inn  sein:.''..  tioiicc  that  the  first  four 
disiinei  d I s 1 1 ilnit  ions , one  when  the  screw  is 
dillcrent.  one.  I'Sr.'i  the.  sci  cw  is  missing.  The  two 
tor  i,:ih  of  Hi  •.-.'.'•rst  three  operators  arc 


I ssi'iit  li  t ly  the 
puss  iln  1 1 1 ICS. 


'nine.  Ihi.y  do  not  d istingiiish  between  the  two 


( xpccti'cl 
1 '>.■/ 

? 3 . 3 

3 2 . 

^ l.f. 

5 .A 

6 .0 

7 .0 


1 1 kp  I ihiMiil 
sOfOni 
; '.'i\,?.i 
yi  1 s 

; ‘5  1 3:.  3 


; > '1 


;>c  ‘i  I 


iMt  i<'ir.  qivon  TiilKL  o hO'I  TIITI’L 

-.1 . 

-:'..(i'j  /(M-l,'. 

-r , 1211  /3? 

-3.‘V‘;  01  /(. 

- . 7U'".'l  '.ii'l 

- . l?O0‘l"b 

s'/rAAo  - .O'i'ii  i3\. 

Ihrsr  cntld  1 1 ion.il  C>'pi:C.ii‘.d 
the.  thi  orcii  thi.t  c.st  Indies 
nar.h  a ci  rtmn  cnnl  idi.ncr 
p.iirs  h.tvr  h•.rn  1 enii.lu: 

/(/(/•  / ikr.  1 ihood  r..l.  ios  . 
si'icily  the  ,1  in  lor  I pr  jli.ili  1 1 i ty  of  tloj  object  heino  there  and 
Hu;  <1. Ml  I'll  C'Hil  nle.'ico  niul  tlu  system  v/ill  estimate  the  number 
(■;  oiurtit  nrs  Hint  v.ill  have  to  be  applied  to  achieve  the 
ih.'sii  nl  cnrii  idi  r.ce. 

Ill  this  situation,  ..  desired  c.onf  idrnce  of  .99  means  that  the 
rst  iri.itr  ol  the  pii)h,ihilitv  that  the  object  is  present  must  he 
above  Ill  Older  to  s.iy  'that  the  object  is  present.  In  order 


/>y 

to 


log-l  ikel ihood  ratios  arc  needed 
the  nui.thcr  ol  operators  reepnred 
notice  that  the  operator /f eaturc 
•.red  according  to  their  expected 
iven  these  ratios,  the  user  can 


to  Sir/  that  the 
pi  oh.ih  I I i l.y  Heit 
a [)l  Kil  I |i|  iilitib  I I H,'/  ol  II. lid 
Ixpecli'il  iiiiiihii  nl  op' r .1  l.ni  ; 
a prim  I pi  iibii  b I I 1 1 y n ! 1 1 .i  lU 

(xpcct'.'d  niiiilu  r ol  c.p' i ni  or.s  ; 
a (iriori  pi  nbab  i 1 1 I y ol  lllii-'i 
txpectod  liiiiiiiir  ol  op/ral  nr ; 
a priori  pi  lib. lb  I 1 I ly  nf  IbliU 
Expected  nui.ber  (if  niiera  Lor ; 
a priori  probabilHy  Cil  1111  111 
Expected  nuriber  o(  ciirratni  s .• 
a pr  iori  prob.ibilH.y  of  111!  hi 
II  Iln:  dnitil'.il 
prohdh  I I I ty  thill, 
the  expected  nunher 


IS 


nut  present,  the  estimate  of  the 


Iln  ohjrrt  is  present 

must 

hn  less  than  ,0], 

li.  ;,  iri-.l  Cl. of  iiii  nee, 

1 

((’  0 

C);.9 

.99 

i 

iv  (in:,  in  .1  t'liil  i.ience, 

(P  0 

C):.9 

.999 

1 

.<  closirid  ci-.of  uleoce, 

(P  e. 

C);.9 

.9999 

(h  ;.  i ri  il  cem  1 i dence , 

1 

(P  & 

C):  .5 

.99 

i 

C,  (h  .sirnl  coni  idr.ncc, 

(P  & 

C):.5 

.999 

f<  .1.  sirrii  co.il  ukneo, 
Ci'i.l  ill.. nee  IS  .{>09, 

(P  f.  C): 
changing 

the  a priori 

Lin:  screw  is  present 

r rom 

. 9 to 

.6  increases 

ol  oper, liars  Iron  one  to  two. 


[Jo  yon  vjant  tfi  save  this  task?  (Y  or  N):Y 
Output  file  iiiiiie : 1 V . 1 Sh. 

An  iiiiiioLaLcd  I isting  ol  this  task 
second  section  ol  this  appendix. 


file  is  presented  in  the 


I 


IV 


iR]At  o'.rcuiiofi  nuE 

The  trio!  cxectitions  provide  o way  for  the  tuer  to  check  the 
overall  performance  of  the  \l\l  program.  He  specifies  the 
desired  confidence  and  the  a priori  prohahil ity  of  the  screv/ 
not  being  present  and  the  system  applies  one  operator  at  a time 
and  prints  the  resulting  estimate  tor  the  probability  that  the 
oh.icct  IS  present. 

1 he  user  can  also  donhlecheck  the  expected  number  of  operators 
required  to  reach  a certain  confidence. 

P 1 c ture ; SC  XCC 1 .PIC 

Oesired  con  I ulonco , (0;tnkc  anythinci  - 1 laCsoluloly  surc);.?i9 
Probability  o(  tlio  object  bcirif)  tborc:.l 

The  probability  ot  tlio  object  111- INC  thirty  .000000 

Since  the  a priori  probability  that  the  screw  is  missing  has 
been  stated  to  he  .1,  the  estimate  lor  the  probability  that  the. 
screw  is  present,  before  any  operators  have  been  applied,  is 
.9.  Ihc  system  prints  the  overall  probability  that  the  scrcvj 
is  present  before  each  application  of  an  operator.  Thus,  the 
sr.qucnce  of  prohahil  it  ics  shows  the  effect  of  incorporating  the 
results  of  each  new  operator. 

OPLRAIOR  VALIIC  1.730060 

The  probability  of  tlic  object  litlNG  there  .094820 
THF  ORJl-CT  IS  lllflU 

Same  tost  picture,  Nov/  one,  or  Quit;  (S,M,or0):S 

Desired  confidence,  (Oitake  an.yttiino  - liabsolutely  sure):. 999 

Ihc  same  trial  picture  is  used,  hut  the  desired  confidence  has 
been  raised  to  .999.  One  operator  was  sufficient  to  reach  a 
confidence  of  .99.  Notice  that  two  operators  are  required  for 
. 999 . 

Probaliility  of  the  object  **1J0T*/.-  beina  there;.] 

The  probability  of  the  oliject  ULltlG  there  .900000 
OPLRATOR  VAl  111'  1 . 730069 

The  probability  of  the  object  BFJNG  there  .994820 
OPERATOR  VAl  lie  1.847515 

The  probability  of  the  object  BEING  there  .999012 
THE  OCJLCl  IS  HURL 

Sane  tcr.t  picture.  New  one,  or  Quit;  (S,N,or0):N 
Picturo:SEXEC?.riC 

Desired  con  f uleiice  , (0:take  anything  -•  1 : ab«.o  lute  ly  sure):. 999 
Probability  of  the  object  *tN01**  being  there:.! 

The  probability  ol  ttic  object  Bi. ING  there  .900000 
OPERATOR  VAl  HI  ' 1 .24, ’’828 

The  probability  of  the  object  BEING  there  .030154 
OPl  RAIOR  VAl  Ilf  1 . 180364 

The  probability  of  the  object  BEING  there  .000321 
THE  OBJECT  IS  I HE  RE 

Same  test  picture.  Mew  one,  or  Quit;  (S,N,or0):0 


EXECUTION  TIME 

At  the.  beginning  of  a day's  run  (or  possibly  more  often,  if 
necessary)  the  program  uses  the  calibration  file,  IV.CAL,  to 
recalibrate  the  camera.  Then  the  program  uses  the  task  file, 
IV.TSK,  to  decide  whether  or  not  the  screv/  is  present  after 
each  attempt  at  picking  up  a screw  from  the  dispenser . An 
example  run  is  presented  below. 

Do  you  want  to  use  the  camera?  (Y  or  M):N 

Display  on  the  syuthesi^'er , halftones,  or  none?  (S,H,or  N):H 
THE  PICTURE'S  CIIANMCI  AND  THE  OVi  RLAY  CHANNEL:  24  25 

Want  to  use  a calibration  file?  (Y  or  N):Y 
Input  file  nanc:lV.CAI. 

Do  you  v/ant  to  use  this  cal  for  an  nuto-cal?  (Y  or  N)Y 
Second  calibration  picture:  SCAL2.f'IC 

SCAIP.PIC  is  the  picture  of  the  calibration  object  taken  by  the 
camera  at  its  current  location.  Given  this  new  picture  and  the 
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IV  1 


Cdlil'roLion  file,  the  sy:>icm  can  autonatically  recalibrate  the 
caricra  hy  applying  the  operators  and  recomputing  the 
transf onaations  from  the  resulting  positions. 

Display  the  calibiation  pictures?  (Y  or  N):N 
Use  a Last  file?  (Y  or  N):Y 
Input  file  name ; IV . TJiK 

Desired  cntitidoncr,  (0:take  iinythino  -♦  1 ;rdisolutcly  surc):.&9 
i’rohability  of  tlie  oh.icct  beinfi  there:.! 

I he  current  task  file,  does  not  contain  the  a priori  prohabil  ity 
that  the  otiJccL  is  missing  or  the  desired  confidence.  1 tie  user 
has  to  specify  these  values  at  the  beginning  of  the  run. 

1'  1 c tore  ;.SI  XI  CJ . I') C 

lui  Ot;di(,  1 i.s  iiiii;i 

P i c tur  c : .St-  XI. (01  . P 1 P 

Itli;  OBdlf.T  I.S  *AtiOlA*  TtlCRC 

Picture :SLXLCb.PIC 

Tlir  OlidfCl  IS  IIIIKC 

PictureiSl  XI  CCi.PlC 

OPLKAIOK  VAI  III  .ohC 

***  AN  UNIXPCntD  VALUE  FOR  THE  OP  *** 

OPERATOR  VAI  LIE  .702 

**A  AN  UNtXPECTLD  VALUE  FOR  HIE  OP  *** 

OPtRATCR  VAI  III  . 711 

***  AN  lINtXPfCllD  VAI  HE  FOR  TtIF  OP  *** 

***  SOMEIHINC  IS  Cl  ORALLY  WRONG  *** 

COULD  NOT  MAKE  A DECISION 

If  an  operator  is  applied  to  a pieture  and  it  returns  a value 
that  is  not  within  three  standard  deviations  of  one  of  the 
expeeted  values,  the  value  is  lahelled  an  unexpected  value.  If 
three  unexpected  values  ate  found  during  the  execution  of  a 
task,  the  program  gives  up  end  decides  that  there  must  be 
something  globally  v/rono.  Consider  figure  II/. 7.9,  which  shovjs 
the.  picture.  SIStCS.I'JC.  Notice  that  the  picture  is 
considerably  darker  and  less  distinct  than  the  training 
liictures.  Such  a picture  may  occur  because  one  of  the  lights 
at  the  workstation  is  out  or  because  the  sensitivity  of  the 
caricra  is  incorrectly  set.  Since  three  unexpected  values  were 
produced  for  this  picture,  the  program  gave  up. 
Picture:SFXFr.7.PIC 
THE  OBJECT  IS  IIIIRE 
Picture : 

End  of  SAIL  execution 


Seel  ion  2 

l ASK  FII.E  l OU  TIIF  SCREW. INSPECTION  TASK 


The  VV  sysieni  out|juis  iwo  types  of  files:  calibirtiion  files  .mcl  task  files.  They  contain 
enough  infoi inarion  to  |)Piforrii  the  S|ii'ciliecl  c.alibi .Hiion  oi  VV  task.  They  contain  a 
Statement  of  the  task,  the  positions  of  the  features,  the  statistical  information,  and  some 
cahbiatiun  inform.iticin,  1 he  formats  lor  the  two  ty|ies  of  tasks  arc  similar. 


1 his  vrcii.iii  lists  the  task  file,  IV  I SK,  pinduccd  at  the  end  of  the  planning  stage  in 
the  trace  pre.-nted  in  the  /ueviniis  section,  The  explanatory  comments  are  surrounded  by 
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parentheses. 


(type  or  tesk) 

^S7;'c  or  interest  operator) 

(size  or  correlation  operator) 

( nunber  or  operator /t catnre  pairs) 


(the  coordinates 
(screen  coordinates ) 

, 5000000  .7^06607 


,5000000 
.^500000 
.4'100000 
.5700000 
.5000000 
.7000000 
.5000000 
. 5000000 


or  the  rcatiircs) 

(workstation  coordinates ) 

^ rt  A r ^ f.  mama 


.•I^OOOOO 
.5000000 
. 5000607 
.A?()(>(w7 
.0700000 
.77160007 
. ITJOCTIO/ 


40  5 /-1 77 
5/77173 
3500750 
4171601 
0405001 
04C4375 
4043507 
3501775 


1 .074  5008 
1.1051401 
1 .0988000 
.8819159 
.9423337 
. 1298235 
1.0272903 
.5763804 


.0000000 

,7500000 

.5000000 

.5000000 

.7500000 

1.0000000 

.0000000 

2.5000000 


( pt  of  interest) 
(lower  It  corner) 
(upper  It  corner) 
(upper  rt  corner) 
(lower  rt  corner) 
(dispenser ) 

( t ip  or  screw) 

( screwdriver) 


0 

.0000000 

.0000000 

.0000000 

.0000000 

(point  of  interest) 

.0000000 

.0000000 

.0000000 

1 

.0000000 

20.0000000 

1.947000 

.2140000 

(lower  left  corner) 

20.0000000 

1.312000 

.1490000 

2 

.0000000 

70.0000000 

2.053000 

.7850000 

(upper  left  corner) 

20.0000000 

1 .752000 

.3110000 

3 

.0000000 

20.0000000 

1 .91  7000 

.3370000 

(upper  right  corner) 

70.0000000 

1.773000 

.3780000 

4 

.0000000 

70.0000000 

1 .870000 

.7060000 

(lower  right  corner) 

20.0000000 

1 .1  79000 

.3670000 

5 

.0000000 

19.0000000 

1.5780833 

.1520299 

(dispenser ) 

20.0000000 

.989007? 

.4479075 

6 

.0000000 

19.0000000 

1 .738000 

.3400000 

(tip  of  screw) 

70.0000000 

1 .190000 

.4030000 

7 

.0000000 

15.0000000 

1 .7740708 

.1829201 

(screwdriver ) 

14.0000000 

1.1985279 

.1500302 

SPLANl.PIC  (the  planning  picture) 

(the  known  and  desired  constraints  & confidences) 


- .3000000 

.0000000 

.3000000 

-.3000000 

.0000000 

.3000000 

- .3000000 

.0000000 

.3000000 

.0000000 

.0000000 

.0000000 

.0000000 

.0000000 

.0000000 

.0000000 

.0000000 

.0000000 

.4857477 

1 .0745698 

.0000000 

.0000000 

.0000000 

.0000000 

.0000000 

.0000000 

.00.00000 

.0000000 

.0000000 

.0000000 

.0000000 

.0000000 

.0000000 

.0000000 

.0000000 

.0000000 

.0000000 

1.0000000 

.0000000 

.0000000 

.0000000 

(the  expected  precision  of  one  operator) 
.0280977  .0733300  .0710070 


(the  calihrat 
23.04449.30-57 
•35.3809800-17 
.0807009 
■22.5000000  -0 
.0(000000 
.0059713  - 

-.0157102  - 

- .0004541 


ion  inf  or mat 
.5208910  99. 
.8524160  152 
.0038601  1. 

.7500000  18. 
.0000000 
,0187518  2. 

.0048718  2. 

.0016353 


ion) 

7404230 

.0100400 

0000000 

7500000 

0000000 

2000957 

3111191 

7957259 


I 
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1V.2 


.0000000  .0000000  .0000000 

.0000000  .0000000  .0000000 


Sl'Clioil  D 

C;a1  IIUIA'I  ION'  WITH  UFSI'KCT  TO  THE  SCREW  DISPENSER 


A c.iriKMa  calibiatioM  and  a VV  location  t.uk  are  very  similar.  Doth  tasks  locate 
features  in  a |iiftinc  of  the  scene  and  use  the  coi ies|icindericc  between  the  screen  positions  of 
the  f^.itures  and  tlie  workst.uion  positions  o!  the  features  to  conijnite  a set  of  parameters.  The 
dificicnce  is  in  the  set  of  par.uiuteis  that  aie  computed.  In  a camera  calibration  the 
paianieteis  define  the  camera’s  position  and  oriciutition  with  respect  to  the  objects  in  the 
scene.  In  a VV  location  t.isk  the  |iai .ameters  s|ieciiy  the  relative  change  of  an  object  from  one 
location  in  the  worksi.ition  coordinate  system  to  another. 


It  would  be  possible  to  extend  the  VV  system  to  compute  the  parameters  for  the 
camera  Howevei.  this  section  docs  not  make  that  extension.  Instead,  it  shows  that  it  is 
possible  to  assume  that  the  changes  in  the  appearance  of  an  object  caused  by  a mislocation  of 
the  camera  aic  clue  to  a change  in  the  location  of  the  object.  This  assumption  is  possible 
because  thcie  is  an  equivalent  change  in  the  location  of  a object  for  each  change  in  the 
location  oi  the  cameia  If  the  changes  are  relatively  small,  it  does  not  matter  which 
assumption  is  m.icic  Thus,  a st.indard  VV  location  ta.sk  can  be  used  to  locate  the  object. 
Having  found  it.  the  screen  positions  of  the  fcaturej  and  the  workstation  positions  of  the 


features  can  be  given  to  the  calibration  routine  to  com|>utc  the  necessary  parameters  for  the 
camcia.  1 his  particular  example  calibrates  the  camera  with  respect  to  the  screw  dispenser. 


Do  you 
Cal Ibra 


Display 
THC  PIC 
Want  to 
S1^e  of 
Si/t*  of 
Automat 


Accept, 
Find  bo 
X and  Y 


want  to  use  the  camera?  (Y  or  M):l'l 
lion  p Icturc iliCAL  1 . PIC 

Consider  rigiirc.  J1/.3.J,  which  shows  the  new  calibration 
picture,  DCALJ.I'JC.  Ihc  pict.nrc.  shows  the  screw  dispenser  on 
the  right  and  the  screwdriver  with  a screw  on  the  end  at  the 
left.  The  programmer  is  going  to  use  the  tour  corners  on  top 
of  the  screw  dispenser  to  calibrate  the  camera, 
on  the  synthcslacr,  halftones,  or  none?  (S,M,or  N):H 
TURE'S  CHAMNEl.  AUD  HIE  OVIRLAY  CHAMNEL:  20  21 

use  a calibration  file?  (Y  or  N):N 
the  Interest  operator 
the  correlation  operator  (e>).  0):0 
ic.  Manual,  or  .Standard  Calibration?  (A,  M,  or  S):M 

Ihc  programmer  uses  the  interest  operator  to  pick  out  the  first 
two  cotners  on  the  screwdispenser  and  then  gives  up  on  it.  He 
points  out  the  final  two  corners  himself. 


Reject, local  ly  adjust, Iry, Coordinate  display, 

St,  or  Sto|)  accepting  suggestions  (A,R,L,T,C,F,  or  S):L 
:*2 
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X iind  Y;>0  +1 
X and  Y;+0  +1 
X and  Y:4] 

X and  Y: 

AcreiJl, Reject, I oca’ iv  t-ijust,lry,Coot  dinate  display, 

Find  host,  or  Stop  a(  cpliiui  sii.niosl ions  ( A, R, L , T, C, F , or  S):A 

The  feature's  X,Y,?:  -.o2b  -.('./b  1.0 

Accop t , Re.iec t , 1 nc«i  1 ly  adjust , 1 ry , Coord i na to  display, 

find  hest,  or  Sto.n  arceptinu  suone.st i ons  ( A, R,  1. , T , C, F , or  S):R 

( A , K , I , 1 , f. , f , or  S ) : R 

(A,R,L,1  ,C,r  , or  S :R 

(A,R,L,l,f,F,  or  S):L 

X and  Y:^:' 

X and  Y:+0  -1 
X and  Y; 

(A,R,l  ,T,C,F,  or  5);A 
The  foaluro's  X,Y,Z:  .625  -.C75  1.0 
A.R,L,T.C,F,  or  S]:R 
A.R,L,1  ,f..F,  or  S :R 
(A,R.L,T,C.f , or  S):S 

Do  you  v/aiil  to  point  out  any  additional  features?  (Y  or  N):Y 
X and  Y for  new  feature  (in  pixels)  (<cr>  to  stop); 100  90 
(A,R.L,1  ,C,F  , or  S):L 
X and  Y;-b  -4 
X and  Y: 

(A,R,L,T,C,F,  or  S):A 

The  feature's  X,Y,Z:  -.625  0 1.0 

X and  Y for  new  feature  (in  pixels)  (<cr>  to  stop):110  70 
(A,R,L,T,C.F,  or  S):L 
X anrl  Y:-10  -20 
X and  Y:40  -2 
X and  Y : ♦ 5 
X and  Y;-l 
X and  Y: 

(A,R,l  ,T.C.F.  or  S);A 

ihe  feature's  X,Y,Z:  .625  0 1.0 

X and  Y for  ncv<  feature  (in  pixels)  (<cr>  to  stop); 

Apiiroxinate  lens  center  (X,Y,7)  ;2b.2b  .T.7b  1C. 25 
TYPE  01  TASk?  (0,  location;  1,  inspection ) ;0 
Si?o  of  the  interest  operator  (4 ,8 , . . . ) ;8 
Si/e  of  the  correlation  operator  (co.  9);9 
known  linits  on  X,  (-dx,+dx) ;-.6  .6 
Knov/n  linits  on  Y,  (-dy,+dy);-.6  .6 
known  limits  on  a,  ( -da,4da) :- 10  10 
Desired  limits  on  X,  f-dx,^dxl ;- . 1 .1 
Desired  limits  on  Y,  ( -dy,4dy);-. 1 .1 
Desired  limits  on  o,  (-dfli,*da);-2  2 
KX:  6.000000  RY:  6.0C0000 

Expected  precision  of  the  correlation  operator,  (on.  1 pixel):l 
Do  you  want  to  determine  the  picture  si^e?  (Y  or  N):M 
X,Y,Z  precision  of  op:  .033397  .023377  .020751 

Do  you  want  to  save  this  calibration  file?  (Y  or  N):Y 
Output  file  name :UlSPtH. CAL 

' H.tvino  dtrinr.d  the  calibration  file,  it  is  possible  to  use  the 

o/icrator /Feature  pairs  to  locate  the  screw  dispenser  at 
execution  time  and  recalibrate  the  camera.  Consider  figure 
IV. 3. P,  which  shows  the  planning  picture  for  the  calibration  on 
the  left  and  a picture  from  a new  camera  position  on  the  right. 
Do  you  want  to  use  the  camera?  (Y  or  N):N 
Display  on  the  syiitlic.s i/er,  halftones,  or  none?  (S,ll,or  N);H 
THE  PICTURE'S  CIlArilJEL  AND  THE  OVLRLAY  ClIARHEL:  20  21 

Want  to  use  a ca  1 iliration  file?  (Y  or  N):Y 
Input  file  nuiicrDl.SIM  H.CAI 

Do  you  want  to  use  this  cal  for  an  auto-cal?  (Y  or  N)Y 
Second  calibration  picture:  DCALZ.PIC 
Display  the  calibration  pictures?  (Y  or  H)'.N 
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rifiiire  l{/.3.3  shows  the  motches  Cor  the  four  features.  These 
results  are  used  to  recalibrate  the  camera.  Having 
recalibrated  the  camera  the  standard  task  file  can  be  used  to 
decide  whether  or  not  the  screw  is  present. 

Use  .1  task  file?  (Y  or  N);Y 

Input  file  nnmc:l\/.TSK 

P1cture:m  XFCl .PIC 

TIIL  OP.JrCI  IS  TIKRE 

Piclurc:lil  XICP.PIC 

lHr  OIIJFCT  IS  **H0U*  THERE 

Picture:t)EXFC3.PlC 


End  of  SAIL  execution 


Scciioii  4 

INTERFACE  TO  THE  MECHANICAL  ARM 


ITiis  srction  biicfly  cIcKiibcs  the  interface  between  the  arm  control  system,  AL  (see 
[FinkrI  75J).  and  the  VV  system  The  puipose  of  the  interface  is  to  make  it  possible  for  a 
|irogiammcr  and  several  processes  (possibly  running  on  several  different  processors)  to  work 
coo|)ei  atively  on  a task.  Finkel  designed  and  implemented  an  interface  called  ALAID  for  the 
Stanford  hand/cye  system  (see  Finkel  [76])  ALAID  provides  several  capabilities  including 
the  lollowing: 

(1)  One  process  can  ask  another  process  for  information,  such  as  status 
information  or  the  value  of  a variable. 

(2)  One  process  can  ask  another  process  to  change  some  of  its  internal 
information,  such  as  the  value  of  one  of  its  variables. 

(3)  1 he  processes  can  use  the  standard  synchronization  primitives,  SIGNAL 
and  WAIT  , to  control  their  interactions. 

1 hrsr  capabilities  are  sufficiently  flexible  to  provide  the  basic  tools  needed  for  a sophisticated 
debugging  system  and  visual  feedback. 

For  visual  feedback  the  system  consists  of  several  different  components  that  have  to 
communicate  with  each  other  in  various  ways: 


(A)  the  arm  and  the  PDPII  that  servos  the  arm, 


Initial i za  t i on 


If  fiiccec&ful 

PRINTC'ALIVE  & ...“) 
Have  tnc  PCiF18  asK  the 
uc.er  hou  many  tries 


PRINT  (■'#  OF  TRIES?") 


TYPE  ("3") 


Park  Arm 


Set  TRIES  to  3 
in  the  PDPll 


PRINT ("TRIES-"TRIES) 
PRINTC'TRYING  TO 
PICK  UP  A SCREU")- 


riove  to  the 
dispenser  and 
try  to  get  a 


Have  the  POP10  check 
for  a scrcu 


Have  the  camera 
take  a picture 


Take  a picture 


Analyze  picture 
Decide  NOT  THERE 


Not  there,  try  again 
PRINTC'TRYING  TO 
PICK  UP  A SCREU" )- 


Move  to  the 
dispenser  and 
try  to  get  a 


Have  the  POP10  check 
for  a screw 


Have  the  camera 
take  a picture 


Take  a picture 


Analyze  picture 
Decide  THERE 


PRINT ("SUCCESS"). 


Continue  the 
assemb I y 


Figure  IV.-l  I 
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(B)  the  camel  a and  the  PDPIO  that  analyzes  the 
pictures, 

and  (C)  the  user. 

The  basic  feedback  loop  is  straightforward-  the  arm  moves  to  the  screw  dispenser  and  tries  to 
pick  up  a screw;  it  stops  just  after  leaving  the  dispenser,  the  camera  takes  a picture,  the  VV 
program  on  the  PDPIO  analyzes  the  picture,  decides  whether  or  not  the  screw  is  present,  and 
tells  the  P1)P1 1 Its  decision;  and  the  arm  cither  tries  again  to  pick  up  a screw  or  moves  to  the 
assembly  to  screw  in  the  sciew.  However,  evcri  for  this  relatively  simple  task  there  is  a great 
deal  of  communication  involved  Consiclei  figure  IV. 4.1,  which  shows  the  flow  of  control 
associated  with  a typical  execution  of  the  P1CK_UP_A_SCREW_TASK. 

The  rest  of  this  section  is  a lisiing  of  the  AL  program  that  performs  the 
PICK_UP_A_.SCR EW_TA.SK,  using  VV  to  check  for  the  screws. 


BEGIN  {screw  checker) 

DEEINE  pick_iip_scrcw  = “MOVE  BARM  TO  chcck_screw" ; 

FRAME  check_screw;  {Tlic  place  where  visual  checking  will  occur) 

FRAME  use_screw;  {The  place  to  insert  the  screw) 

ROT  r;  {A  rotation  variable  to  represent  "straight  down") 

SCALAR  type;  {TYPE  holds  the  typo  of  signal  being  sent  from  the  arm 

program  to  the  vision  program,  eg.  to  say  "I  AM 
READY  FOR  YOU  TO  CHECK  THE  SCREW"  the  AL  program 
puts  a "2"  in  type  and  signals  SAIL  ...  when  the 
SAIL  routine  finishes,  it  signals  AL  so  the  AL 
program  can  continue  ...  the  codes  are: 

0:  not  set 

1:  do  the  initialization  (set  TRIES,  etc.) 

2:  check  for  a screw  and  set  THERE  (1  is 
true) 

3:  ask  the  user  to  place  the  screw  on 
manual ly 

A:  success,  a screw  is  on  the  end) 

SCALAR  tries:  {Specifies  the  max  number  of  tries  to  get  a screw) 

SCALAR  there;  {Variable  set  by  the  screw  checking  routine  to  indicate 

v/liethcr  the  vision  routine  thinks  the  screw  is 
there  or  not,  a 1 (actually  anything  greater  than 
0)  indicates  that  the  screw  is  there) 

EVENT  sail;  (This  is  the  universal  event  used  to  signal  the  SAIL  routine 

that  something  is  to  be  done) 

EVENT  al;  (The  universal  event  used  by  the  SAIL  routine  to  tell  the  AL 

program  to  continue,  the  SAIL  part  has  been  done) 
SCALAR  k;  {A  loop  variable) 


Paf.e  2(>S 


IV.-l 


r - ROT{YIIAT,  U’-O ) ; {Set  r to  "straight  down") 

{Define  the  two  frames  in  terms  of  r) 

clieck  screw  •-  (KA!'il(r,  VI  C101'(  SO  .OA,  Au.O/,  0.16)); 

iiso_scrow  ^ FKA.’II(r.  VtC70K(30 .00,  35.00,  8.00)); 

{Print  soirielhin'i  to  incliciitc  you  arc  alive  and  park  the  arm) 

PR1NT(  "Al  IVC  Atilt  r,l  SlICULAlltJC"); 

MOVh  RAKM  10  BPAHK; 

{WITH  DURATION  ? 2.0*SrC) 

type  •-  1;  {Ask  the  SAIL  routine  to  initialize  things  ...  eg.  tries) 

SIGNAL  sail; 

{Ixpcct  the  SAIL  routine  to  a.sk  tiie  user  appropriate 
(|iicst  ions,  sot  the  associated  variables  (using  SETVAL),  and 
then  return  control  to  this  AL  program) 

WAIT  al; 

PRINT  ( "tr ies" , tr  ics ) ; {Just  to  doublecheck  that  the  value  was  set) 


tlicrc  •-  0; 
k - 1 ; 

WIlUt  ((k  < tries)  and  (there  *■  1))  00 
111  GIN  { pick  up  loop ) 
k *•  k 1 ; 

. ASSERT  r.ARil  = 11  PARR; 

PRINT ("TRYING  TO  Cl  I A SCRFW"); 

pick_up_scrcw;  (A  macro  that  rioves  the  arm  to  the  dispenser  to 

pick  up  a screw.  The  arm  stops  just  after 
leaving  the  screw  dispenser.) 

MOVE  BARM  TO  check^scrcw; 

typo  ► 2; 

SIGNAL  sail; 

(Have  the  SAIL  program  take  a picture  and  analyze  it  to 
decide  whether  or  not  the  screw  Is  present.  If  it  is 
there,  have  the  program  set  the  variable  THERE  to  one.) 

WAl  I al ; 

END;  {pick  up  1oo|)) 

ASSERT  BARM  = chcck_scrcw; 

{Now  sec  what  the  results  v/erc  ...  either  the  vision  routine  decided 
that  the  screw  was  there  and  set  THERE  or  the  program  ran  out  of 
tries) 

{If  the  scf cw  is  still  not  on  the  end,  pause  and  ask  the  user  to  put  it 
on  the  end) 

IF  there  * 1 IHI  N 

P.FGTN  {screw  not  on  yet) 

PRINT(  "PHASE  LET  ME  HAVE  A SCREW"); 
type  •-  3; 

SIGNAL  sail; 

{(hive  tlio  SAIL  program  make  sure  that  llio  user  places  a ^ 

I 
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screw  on  the  end  of  the  screwdriver  and  gets  the  arm 
ready  to  proceed) 

WAIT  a); 

END;{scrcw  not  on  yet) 

(Use  the  screw  and  then  continue  with  the  rest  of  the  assembly) 
MOVE  BARM  TO  use_screw; 

PR1NT( "SUCCESS! " ) ; 

END:(scrcw  checker) 


Since  only  one  or  two  conelaiion  0|iciatois  are  required  to  decide  whether  or  not  the 
sciew  is  on  the  end  ot  the  .sciewdrivei.  the  time  to  take  the  picture  and  the  time  to  send 
messages  back  and  forth  between  the  processors  dominate  the  VV  time,  i e..  the  time  to  locate 
the  matches  and  conij^ute  tlic  probabilities  I he  overall  elapsed  time  between  the  AL  request 
and  the  answer  is  appioximately  two  seconds. 


Prtoe  IM" 


V 


APl'r.NDlX  V HOLE  L.OC.AT  ION  EXAMPLE 


1 In',  st  fiimi  IS  ti  .ii  H of  H |iiiii^;i,iiiii;H  i (It  H \'V  |jrogram  to  locate  a screw  hole. 

At  proj'.i .iminiiig  time  the  cx|icctecl  (irccisinii  cl  the  correlation  Ojacrator  is  used  to  determine 
the  1 esolntinii  ii>|iiiied  to  inoduce  the  cii  t-iii.d  im  ..isiun.  'Lite  iiaining  stage  demonstrates  the 
ciilhiig  of  1.1, id  m.itche'  '1  he  (>1. inning  stage  i.inl.^  the  o|iei atot/feature  pairs  according  to 
tin'll  asiiHge  siiiicnii.tl  coiisisii.ncy.  C>|ii.iaio;s  ih at  fiiici  their  features  at  locations  that  are 
consistent  with  the  othei  m.<iche'  are  eiven  -•u  id  i.itin  s 1 he  execution  stage  denionsti ates 
the  use  of  picvinn,'.  ni.aiehcs  to  leduce  the  tolc  i.mce  re  iuii  about  a new  feature  to  be  found. 


I'ltort’/.ttitih'f.  1 n:r 

Po  you  waul  tC'  use  the  ciiuC'i  a’  (Y  or  i;):U 
Cci  1 ibi'ci 1 1 on  I)  1 c lure  tV.'CAt  1 .Pie 

Displiiy  on  Itic  svii I tie.s  i zer  , tni  I f tones , or  none?  (5,11. or  M):H 
im  PiCTUlU  '5  CIIAi;!JI  1 Alii)  Till  OVI  Itl  AY  CIIA'fl.'L  I : 24 

Want  to  use  a ca  1 ihr.it ion  file?  (Y  or  fiJiIJ 

C<)lil'rcitc  the  Cc'iiticra. 

Siito  of  the  interest  oncrator  (4  .ft,  . . . ) 

Si^c  of  the  correlation  operat.or  (eg. 

Autonatic,  M.antm  I , or  StanclartI  C.i  1 ilir.it  ion  ? (A,  M.  or  5):M 


Point  out  the  features  in  the  calibration 
Accept , Reject . I oca  1 ly  adjust , Iry, Coot  cl  iiuite  cli.splay , 
find  be.st,  or  5tot»  accepting  soonest  ion.'.  ( A,  R , L , 1 , C,  I , 
X and  Y:+l  +1 
X and  Y:+l 
X and  Y: 

Accept, Reject, local  ly  adjust,  Iry , Coord  inat.e  display, 
rind  be.st,  or  Stop  acceptinn  suociest.ions  ( A,  R , I , 1 , C , F , 
Ihc  fc.iturc's  X,Y,/:  2.0  0 0 

Acce()t , Ru  jecl , 1 oca  1 ly  ail  jus  t , Iry , Coord  ii.aLc  d i splay. 
Find  best,  or  Stop  accepting  sunoostions  ( A, R , L , T , C, F , 
X and  Y;+3 
X and  Y;+l 
X and  Y: 

Accept , Re  jnc  t , I oc.i  1 ly  adjust, Iry , Coord ina to  display. 
Find  best,  or  Stoj)  acccptinn  sucincstions  ( A,  R,  L,  T ,C,F  , 
The  feature's  X,Y,/;  1.5  .50 
(A.R,L.T,C,1  , or  S):l 
X and  Y:*0  ♦! 

X and  Y: 


(A,R,L.T.C,F. 
Ihc  feature's 
{A,R,L, T.C.r  . 
The  feature's 
(A.R.L.T.C.f . 


or  S):A 
X,Y,2:  .5  .5  0 
or  S):A 
X,Y,/:  000 
or  S):L 


X and  Y:-2 
X and  Y:-l 
X and  Y: 


picture 
or  S);L 

or  S)  ;A 
or  S) : L 

or  S)  :A 
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(A,R.l  .T.C.P  , 
The  feature’s 

(A,R.I .i.c.r. 

The  feature's 
(A.R.L.l.C.f  . 
X and  Y : 1 - 6 
X and  Y;+ 1 
X and  Y: 


or  S);A 
X.Y.Z:  0 2.0  0 
or  S):A 

X.Y.Z;  .b  1.5  0 
or  S)  :L 


(A.R.t  .T.C.P  . or  S):A 

Ihe  feature's  X,Y,Z:  2.0  2.0  0 

(A.R.l  .T.C.F,  or  S):S 

Do  you  want  to  point  out  any  additional  features?  (Y  or  N):N 

figure  V.I  ihni'.'s  yCAlI.PlC  and  the  set  of  reatures  defined  by 
the  preceding  process. 

Approximate  lens  center  ( X , Y . / ) : . /f)  -12.5  13 
lYPf  or  TA.SR?  (0,  location;  i,  inspect  ion ) :0 
Size  of  the  interest  operator  ('i.0,...):8 
Size  of  the  correlation  operator  (eci.  <j):9 
Known  limits  on  X,  ( -dx  . •*  dx  i ; - . 6 +.o 

Knov;n  limits  on  Y,  ( -rly,  + dy  i . 1 1 

Known  limits  on  o,  ( -d>i , +da  ) : - 1 5 ♦lb 

Desired  limits  on  X,  ( -dx , +dx ) : - . 0 j +.05 
Desired  limits  on  Y,  ( -dy , +dy  1 : - . O’j  +.05 
Desired  limits  on  a,  ( -da.+do) ; -3  +3 
RX:  12.000000  RY:  2.000000 

Expected  precision  of  the  correlation  operator,  (cf).  1 pixel  ):1 
Do  you  want  to  determine  the  picture  size?  (Y  or  N):Y 


bcteri'iinc  the  location  and  resolution  of  the  camera:  (1)  use.  a 
sample  set  of  features  to  determine  the  tolerance  i/olume  of  the 
task,  (?)  project  that  volume  onto  the  screen  to  form  the  task 
tolerance  region,  (3)  overlay  the  task  tolerance  region  on  the 
planning  picture  on  the  left  (d)  scale  the  region  to  indicate 
v.'hethcr  the  camera  should  he  roomed  in  or  out  to  produce  the 
desired  resolution , (b)  show  the  scaled  task  region  on  the 
right  side  of  the  display,  and  (6)  zoom  the  camera 
appropriate ly. 

Planning  p icturc  :WPl ANl . PIC 


Point  out  a sample  set  of  features  (to  define  the  task  volume). 
X and  Y for  new  feature  (in  pixels)  (<cr>  to  stop):70  200 
(A.R.L.l.C.F,  or  S);L 
X and  Y:+b  -20 
X and  Y;4A  -A 
X and  Y; 


(A,R.l ,1 ,C.f  , or  S):A 

Input  the  feature's  hcuiht  ahovc  the  plane,  ic.  Z:0.0 
X and  Y for  nev/  feature' (in  pixels)  (<cr>  to  stop):110  200 
(A.R.L  ,1  ,C,F  , or  S);L 
X and  Y:+0  +12 
X and  Y: 

(A,P.L,T,C,F,  or  S):A 

input  the  feature's  height  above  the  plane,  ie.  Z:-.5 
X and  Y for  new  feature  (in  pixels)  (<cr)  to  stop):200  110 
(A.R,l,l,C.f,  or  S):L 
X and  Y:-7  +8 
X and  Y: 

(A,R,L,T.C,1  , or  S):A 
2:  .25 

X and  Y for  new  feature  (in  pixels)  (<cr>  to  stop):165  105 
(A.R.L.l.C.f,  or  S):L 
X and  Y:-5  +12 
X and  Y: 


(A.R.l .1 .C.F , or  S):A 
7:1.32 

X and  Y for  new  feature  (in  pixels)  (<cr>  to  stop): 

Approximate  screen  coords  of  the  point  of  Interest  (x,y):100  200 


V 


Paf'e  213 


X and  Y:  + 15  -H 
X and  Y; 

The  I of  the  point  of  .ntrrost,  7:0.0 

higiire  M .?  shows  yPI  nil]  .hlC  and  the  samp''e  set  or  features  used 
to  define  the  tolerance  volume  for  the  task. 


Or.tcrmine  the  change  in  resolution . 

RfSOLUTION  or  ONh  PIXLL  A1 OIJG  IHf  SCRfEN'S  X AXIS: 

(I'lANE'SX:  .0?fl701  PlANfS  Y:  -.010617) 

RESOLUTION  OF  ONE  PIXEL  A1 ONG  HIE  SCPnEEN'S  Y AXIS: 

(PIANL'SX;  -.010065  PLANE'S  Y;  -.035794) 

X.Y.Z  prcci.'ilon  of  op:  .0.''5177  .034902  .033167 

HIE  'ZOOM'  lAClOR:  .69E’.04? 

4 simple  heuristic  is  used  to  determine  the  ^oom  factor 
regthred  to  achieve  the  desired  resolution  along  each  of  the. 
axes  in  the  cal  ihration  plane;  given  the  expected  precision  of 
one  operator  and  the  desired  precision  about  the  point  of 
interest,  the  mom  factor  is  the  ratio  of  the  expected 
precision  over  the  desired.  The  ?oom  factor  for  the  whole 
picture  IS  the  maxiniim  ot  the  two  zoom  factors  along  the  X-axis 
and  the  Y-axis.  This  overall  zoom  factor  guarantees  that  both 
of  the  desired  precisions  will  be  met.  It  is  a conservative 
estimate  of  the  actual  resolution  required  because  it  is  based 
upon  the  expected  precision  of  one  operator  instead  of  the 
expected  precision  from  several  operators , which  would  he 
greater.  As  mentioned  in  section  0.6  it  would  be  possible  to 
estimate  the  necessary  resolution  more 


some  example 
rc.solut  ions . 

THE  ORIGINAl  WINOOW  IS: 

(MINX.MAXX):  45.920710 

(MINY.MAXY):  73.201710 

THE  R1  Sill  I INC  WINDOW  IS: 

(MINX.MAXX):  75.542129 

(MINY.MAXY):  90.352409 


location  tasks  with 


244.379760 

250.164150 


accurately  by  performing 
pictures  at  different 


214.069290 
213.079620 

figure  17.3  shows  the  task  tolerance  region  overlayed  on  top  of 
PLAN]. PIC  and  the  scaled  tolerance  region  or  the  right  of  the 
display. 

R:12. 000000  M:3  OPERATOR  SIZE : 9 

THE  MINIMUM  AMOUNT  Of  CORE  TOR  PICTURES  IS:  2844  (36-BIT  WORDS) 

Ihis  numher  is  the  total  amount  of  core  required  to  hold  the 
pictures  used  in  this  1/i/  task.  It  is  based  upon  one  4-bit  test 
picture  and  the  portions  of  the  planning  picture  required  by 
ten  correlation  operators. 

Nov/,  zoomed  picLiire:WI'lAN2  .PIC 
Satisf iod?  ( Y or  N) :N 
New.  zoomed  piclu/T;WI'LAN3.P]C 
Satisf led?  ( Y or  N) :Y 


Recalibrate  the  camera  at  this  new  focal  length. 

Automatic  or  Manual  Calibration?  (A  or  M):A 

New  calibration  picture:  WCAL2.P1C 

figure  11.4  shows  UCAll.PIC  on  the  left  and  the  new  calibration 
picture  on  the  right.  1 he  calibration  object  appears  to  he 
larger  (i.e.,  at  a higher  resolution)  on  the  right,  but  it  is 
not.  After  the  focal  length  of  the  camera  has  been  decreased 
(decreasing  the  resolution  of  the  picture  and  the  size  of  the 
appearance  of  the  calibration  object),  the  portion  of  the 
picture  outlined  on  the  right  in  figure  I/. 3 is  extracted  and 
expanded  so  that  it  fills  up  the  same  portion  of  the  display  as 
the  original  picture.  Notice  that  the  precision  listed  below 
is  LOi'/LR  i.e.,  less  precise)  than  the  precision  mentioned 
earlier. 

Display  the  calibration  pi<.tures?  (Y  or  N):Y 

Use  the  opcralor/f cature  pairs  defined  for  the  calibration  to 
recalibrate  the  camera.  That  is,  apply  the  operators  and 
locale  a stiff  icient  number  of  the  features  to  recompute  the 
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col inention  matrix. 

X,Y,/  precision  of  op;  .O'JIBIS  .050177  .047707 

Do  you  want  to  save  this  calibration  file?  (Y  or  N):Y 
Output  file  narie:\/.CAL 
COLINCATION  MATRIX: 

3P.9790P10  -b.?200699  RG. 1000000 

-9.140747?  -21.9564100  178.1000000 

.0039588  .0320065  1.0000000 

Use  a task  file?  (Y  or  N):W 

Choose  the.  potential  operator/f  eaturc  pairs:  (1)  apply  the 
interest  operator  to  l^l'iANJ.PIC  and  (2)  select  the  promising 
pairs . 

1 r a 1 n inn  Picture :WTf  ST  1 .PIC 

Accept , Reject , Loca  1 ly  adjust, Try, Coordinate  display, 

Find  best,  or  Stop  accepting  suggestions  ( A, R, L, T ,C , F , or  S):L 
X and  Y:-7 
X and  Y;-) 

X and  Y:-0  *1 
X and  Y: 

Accept, Reject, Local ly  adjust, Try, Coordinate  display, 

Find  best,  or  Stop  accepting  suggestions  (A,R,L,T,C,F,  or  S):A 
Input  the  feature's  height  above  the  plane,  ie.  Z:0 

figure  l/,6  shows  the  new  planning  picture,  WPLAN3.PIC . on  the 
left  and  a lest  picture  on  the  right.  The  first  feature  is 
shown  on  the  left. 

Accept , Reject , I ocal ly  adjust, Iry, Coordinate  display. 

Find  best,  or  Stop  accepting  suggestions  (A,R,L,T,C,F,  or  S):R 
(A,R,L,T,C.F,  or  5):L 
X and  Yr+JO  -5 
X and  Y:49  -4 
X and  Y; 

(A,R,L,T,C,F,  or  S);T 
VALUt  .884 

(A,R.L,T,C.F,  or  S):A  , ^ 

Input  the  feature's  height  above  the  plane,  1e.  Z:-.5 


(A.R,L,T,C,F , 

or 

l:R 

(A.R,L,1 .C.F, 

or 

si 

1:1 

VAI  UE  .820 

(A.R,L,I.C,F, 

or 

S):A 

Z:1.32 

(A,R,I  ,1  ,C,F , 

or 

l:R 

(A,R,l ,1.C,F , 

or 

|:R 

(a,r,l,t,c,f. 

or 

s 

l:R 

2^’ 3^^ ’ ^ ^ ’ 

(a;r,l,i.c,f. 

or 

5 

l:A 

or 

S):L 

X and  Y;+0  -3 

X and  Yi+lO 

X and  Y:-10  - 

10 

X and  Y;42 
X and  Y:+0  flO 
X and  Y: 

(A.R,L,T.C.F,  or  5):T 
VALUt  .740 
(A,R,L,1 ,C,F,  or  S):A 
ZrO.O 

(A,R.L,T,C,F,  or  S):R 
(A,R,L.T,C,F.  or  S):S 

Do  you  want  to  point  out  any  additional  features?  (Y  or  N):Y 
X and  Y for  new  feature  (in  pixels)  (<cr>  to  stop):20  100 
(A,R,l ,T,C,F,  or  S):+0  -15 
****UNRf COGNIZED  ANSWER*** 

(A,R,L,I,C,F,  or  S):L 
X and  Y:40  -15 
X and  Y:40  -4 
X and  Y: 
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(A.R.L.l.C.F,  or  S):A 
7:0.0 

X cinJ  Y for  tiov/  ft'iTlurc  (in  pixels)  (<cr>  to  stop):160  35 
(A.R.L.T.C.r.  or  S):L 
X .mil  Y;-10 
X ntui  Y:-7 
X .inO  Y:-t0  +1 
X .tiul  Y: 

(A.R.L.T.C.r,  or  S):T 
VAllIL  .Aoa 
(A.R.L.T.C.r.  or  S):A 
7 : .75 

X .nnil  Y for  now  fo.ilure  (in  pixels)  (<cr>  lii  .slop): 
fipproxmuilo  sti  cen  coords  of  the  point  of  interest  (x,y):45  100 
X .-me)  Y:+M  -3 
X .inci  Y:  + 0 -1 
X .mil  Y: 

The  7 of  llm  point  of  intcrosL,  7:0 

//i»A  conp'/etes  the  /ooprrtW.'vmp  time.  The  camera  has  been 
fio.'i  I L ioned  and  ca  I ibrat'.d , the  task  has  been  stated,  and  the 
itoLr.nLial  ot>eraLor/t  catiirc  patrs  have  been  defined.  Figure  [/.6 
.shows  the  features  that  have  been  chosen, 


iRAiiiiNi;  iiiu: 

Apply  the  operators  to  .several  training  pictures  and  gather 
statistics  on  the  structural  consistency  of  the  matches.  All 
of  the  operators  arc  applied  to  each  training  picture.  Their 
natches  arc  handed  to  the  1 cast-souares  culling  routine,  v/hich 
determines  the  bcr.t  fit  and  culls  out  the  matches  with  large 
residual  errors.  Ihe.  culling  is  done  in  two  steps;  (1) 
permanently  cull  any  match  that  is  significantly  inconsistent 
(i.c.,  more  the  three  standard  deviations  away  from  its 
expected  position) , and  ('/)  temporarily  eliminate  the  worst 
rcm.iining  match  and  try  another  fit;  if  the  fit  is 
s ignif icantly  better  than  the  one  including  the  match, 
permanently  cull  the  (/uesL  ionahle  match;  if  the  fit  is  not 
signif icantly  better,  leave  the  questionable  match  in  the  set 
of  good  matches  and  slop  trying  to  cull  matches. 

Training  P icliirr  : W 1 1 S 1 1 . P]  C 

Do  you  want  to  iric. luilo  the  re.siilts  of  this  trial?  (Y  or  N):Y 
THE  WORST:  o .23330)  ...  Al'.-SOLUIL 

Jhc  natch  for  operator  6 is  so  far  away  from  the  position 
ptcdictcd  by  the  best  fit  that  it  is  immediately  culled  from 
the  set  of  matches  for  thi.';  training  picture.  Figure  V.7  shows 
the  match  for  oiir.rator  o.  notice  that  the  correct  match  for 
operator  0 is  right  ou  the  i dgc  of  the  picture.  For  some 
reason  the  operator  did  not  find  it. 

THE  WORST:  5 .0)5230  ...  UtnATIVE 

OLD  AND  NIW  TOl FRANCIS  ON  a.dx.dy: 

.7)115?  .70.1760 

.00.3005  .003862 

.005012  .005281 

TOLERANCES  DID  not  IKPKOVL  CY  DLLEliNG  IIIAl  POINT 
NO  IMPROVLHLNl 

Ihi:  m.itch  for  operator  6 i".).',  the  worst  remaining  match,  but 
taking  it  out  of  the  set  did  not  significantly  improve  the  fit, 
.so  it  was  left  in  the  set. 


1 ■ 

* 

PLANNED 

POSH  ION 

81  SI 

LSllNAll. 

I'UASUREO 

1 

2.0145H 

.006150 

I.G30305 

. 100/88 

1 .028)04 

.094524 

.006031 

-.703600 

-,.379378 

-.73721? 

-.380342 

-.240852 

' 3 

1.340150 

-.ORIO.TS 

.000153 

-.050308 

.061510 

-.049120 

A 

,405959 

.075190 

.I0<3203 

.073312 

.103057 

.686125 

1 

: 5 

,701500 

-.005650 

-.000140 

-.032055 

-.083456 

-.036095 

-.045670 

.460445 

-1.00037? 

.545085 

-.737630 

.606142 

1 7 

2,545040 

,842800 

2,101587 

.832145 

2.190081 

.828087 

f «+ ■»•  i.'h  ,«(iii((g|iir 
.,■■1 ..« li ' 


Figure  V.7 


Another  trial?  (Y  or  N):Y 
Training  1“  ic tore  :W  1 t.S12  . PIC 

Do  yon  v/ant  to  Includo  the  re  .nits  of  tins  trial’  (Y  or  N):Y 
IHC  V/ORSl  : 6 ...  IllMAlIl/i: 

OLD  AtJI)  NIW  TOIIRANCIS  OU  a.dx.dy; 

^.OlDii.lO  ..TnOC?/! 

.nh3  70'1  .00C<df)l 

.0  hoc  13  .00C67  7 

1MPROVLMLN1 

Ibis  tirir  the  r.uitrh  for  operaior  C ivas  not  far  enough  av/ay  fron 
the  imsition  pr (iiictc.d  hy  the  fit  to  he  immediately  culled. 
lloeTver , It  the  t'orsL  match  and  leaving  it  out 

.significantly  improved  the  fit,  so  it  was  removed  from  the  set 
of  consistent  m itches. 


tiir 

WORST : 

/ 

.o;(iC4s 

UNTATIVE 

OLD 

ANii  Nl  W 

TOl  FRANCES  ON 

a , d/ , dy : 

.3u96''l 

.439893 

.006961 

.006392 

.008677 

.00/696 

NO 

IMI'ROVI  Ml  NT 

0 

PI  ANNl  1) 

POSH  ION 

111  ST 

ESI  IMA U 

Mf ASUREO 

1 

2.014514 

.096158 

1.7353.'i4 

.193656 

1.735452 

.184310 

2 

.006031 

-.293669 

-.253491 

-.286197 

-.242347 

-.293800 

3 

1.340150 

-.083038 

1 .069752 

-.01 '.81  a 

1 .071208 

-.015020 

4 

.465959 

.6.-  5199 

.164462 

.652208 

.153790 

.675135 

5 

.281509 

- .085656 

.012310 

- .rM..'253 

.036402 

-.061121 

6 

- .645620 

.466445 

-.93881 1 

.443405 

-.715465 

.976305 

7 

2'.  54  5940 

.042806 

2.232491 

.963547 

2.206353 

.951339 

Another  trial?  (Y  or  N):Y 
Training  Pic ture :W 1 LS 1 3 . PIC 

Do  you  want  to  include  the  results  of  this  trial?  (Y  or  N):Y 
THE  WORST;  2 .2CC10C  ...  TLMTATIVE 

OLD  AND  NEW  TOLERANCI  S ON  a.dx.dy: 

2.201107  l.bC'1930 

.OA0013  .032001 

.Oh.T.Tuf)  .039083 

IMPROVrtTENT 

THE  WORST:  0 .102011  ...  UNTATIVE 

OLD  AND  NIW  TOLtRAHClS  ON  o.dx.dy: 
l.bP.403o  1.002008 

.032001  .010835 

.0;'.''<f,.'’.3  .020^01 

IMPROVrilENT 

THE  WOP.SI:  3 .0008.12  ...  UNTATIV’E 

OLD  AND  NI  W TOIERANCI.S  ON  a,d>:.dy: 

1.002008  .783/12-1 

.010835  .0MO25 

.020-101  .021650 

NO  IMPROVIHENT 

0 PI  ANNI  I)  P0.SJ  I JON  RISl  E51INAU  MEASURED 

1 2.0H5I-1  .0'i61  58  1.333192  -.0-11914  1.300780  -.030977 

2 .000031  -.293009  -.711028  .023521  -.403734  -.109135 

3 1.340150  -.083038  .635842  -.009025  .673031  -.013433 

4 .465959  .025199  -.O'-CIOO  .810510  -.03-1034  . 770823 

5 .281509  -.085056  -.390002  .1047/9  -.395031  .163998 

6 -.045620  .400445  -1.17/125  .90993/  -.964250  1.090785 

7 2.545940  .842806  2.017903  .564240  1.981370  .535230 

Another  trial?  (Y  or  N);Y 
Iroining  Picture :Wll. 5 14 . PIC 

Do  you  want  to  include  the  results  of  this  trial?  (Y  or  N):Y 
THE  WOR.ST:  5'  .019895  ...  ILNTATIVE 

OLD  AND  NEW  TOLERANCES  ON  o.dx.dy: 

.263023  .263642 

.005411  .005/17 
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.00602? 

.006472 

TOLERANCES  DIO 

not.  IMPROVE 

BY  DLIEIING  THAT  POINT 

NO 

INI'ROVIMENT 

# 

t'l  ANNl  1) 

POSH  ION 

IN  ST 

ESIIMATE 

MEASURED 

1 

2.014014 

.096158 

2. 100502 

.084181 

2.107497 

.091572 

2 

.006031 

- . 2 9366E# 

.135249 

- .200199 

.123865 

-.216194 

3 

1 .3401  00 

- . 083038 

1 .478126 

- .001565 

1 .461673 

-.044214 

4 

.460000 

.625199 

.(>69897 

. 688888 

.054979 

.085162 

5 

.281000 

- .085656 

.4;  004.3 

- .012014 

.402023 

- .003581 

6 

- .64  0620 

.4i.(.44  5 

-.4/8155 

.585102 

-.471110 

.571449 

7 

2 . 040640 

.842806 

2.7280/5 

.803730 

2.744815 

.797935 

Another  trial? 

(Y  or  N):Y 

Training  R i cturc : W1 1 ST  b . I'lC 

Do 

you  want 

to 

inclutle  the 

re. Ml  Its 

of  this  trial? 

(Y  or  N):Y 

THE 

WORST : 

1 

.037/7? 

TENTATIVE 

OLD  AND  NfW 

lOLERANCrS  ON 

a.dx.Uy ; 

.404727 

.400978 

.008494 

.008277 

.009000 

.0082/9 

NO 

IMRROVf HENT 

* 

PLANNED 

POSH  ION 

BEST 

ESTIMATE 

MEASURED 

1 

2.014014 

.096158 

2.306920 

.239000 

2.404679 

.240593 

2 

.006031 

-.293609 

.403978 

- .337308 

.373446 

-.317529 

3 

1 .3401  00 

- .0830  38 

1 .712388 

- .002191 

1 .698851 

-.008702 

4 

.460909 

. 62  5 1 99 

.775488 

.020728 

.746931 

.634586 

5 

.28 1009 

- .085050 

.6586R1 

- .104319 

.675836 

-.114726 

6 

- .64  0620 

.400445 

-.316244 

.358179 

-.309320 

.338962 

7 

2.045940 

.842800 

2 .825802 

1.032906 

2.836590 

1 .034477 

Another  trial? 

(Y  or  N):N 

EINAL  STATISTICS 

# 

N 

Ml  AH 

Sfi 

1 

5.000000 

. 0 i 9088 

.01299.3 

2 

5.000000 

.081760 

. 142108 

3 

5.000000 

.023548 

.025505 

4 

5.000000 

.020411 

.01.3491 

b 

5.000000 

. 0 1 6326 

.008771 

C 

5.000000 

.2.39  72  7 

.2.33803 

7 

5.000000 

.021689 

.010042 

I Notice  LUiit  the  steti.stics  for  each  operator  include  the 

results  from  each  trial  {i.c.,  N-5J  even  though  some  of  the 
I matches  were  culled  from  the  best  fit.  The  worst  matches  vtcre 

t culled  so  the  best  possible  fit  could  be  determined.  Then  the 

rcsidu.ils  for  all  of  the  matches  were  computed  and  added  to  the 
statistical  information . In  this  way,  the  operators  that  match 
; the  mong  features  (and  hence  are  far  away  from  their  predicted 

’ positions)  produce  large  values  for  their  average  residual 

1 error. 


PlAMUltK,  TIME 

Ihc  ranking  scheme  simply  orders  the  operator/feature  pairs 
according  to  their  average  residual  errors. 

THE  ORDERED  SET  OF  FEATURES  ...  # aiul  the  ranking  value 

5 .016326 

1 .01  DORR 

7 .0216R0 

3 .023048 

4 .026411 

2 .OR1760 

6 ,230727 

Ihc  system  renumbers  the  features  according  to  this  ranking. 
Ihus,  from  nc>w  on  the  old  number  five  will  be  number  one,  etc. 
Do  you  want  to  save  this  task?  (Y  or  N);Y 
Output  flic  namo:V.TSk 
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fvrriiiiofj  7?Mr 

Ihrcc  cxi)mi>lc  cxccuLion-t  inc  performances  arc  presented  hclor/. 
In  the.  first  one  the  first  three  operators  locate  the  desired 
hole  sufficiently  precisely;  in  the  second,  four  operators  are 
nrrdcd;  ami  in  the  third,  the  operators  can  not  produce  the 
<h  sired  precision. 

Piclufc:WLXCCl .PIC 

Do  you  WciiiL  to  S' e ciitli  fcnltiro's  nutcti  Jis  it  is  found?  (Y  or  N):Y 
***  for  drliiKi'iinu  ...  sot  liypfllj  to  V.  (i;  f.  V): 

h'tl‘[...]  hnids  the  kiiovm  end  desired  tolerances  for  the  task. 
Ill  is  sLaterieni  provides  I'My  for  the  user  to  temporarily 
change  the  these  values  m order  to  see  hov/  vjcll  the 

opcraLor/l  caiiirc  pairs  hcliave  in  different  situations . For 
this  particular  cjier.iition  ol  the  task  no  changes  were  made. 

How  many  features  .should  the  locolion  routine  start  with:  3 
<cr>  to  continiio: 

1 he  system  shovjs  all  of  the  features  on  the  planning  picture 
and  asks  for  a carriage  return  to  indicate  that  the  user  has 

seen  them  and  is  ready  to  apply  one  operator  at  a time.  The 
system  also  shows  the  match  for  each  operator  and  pauses  again 
if  the  user  specified  that  ha  wanted  to  see  each  feature's 

match . 

VALUE  .856 
<cr>  to  continue: 

VALUE  .054 
<cr>  to  continue: 

VALUE  .921 
<cr>  to  continue: 

I igtirc  t/.8  shows  the  planning  picture  on  the  left  and  the 

execution-time  picture  on  the  right.  The  three  features  and 
the  corresponding  matches  are  outlined. 

THE  WORST:  2 .014804  ...  UMTATIVE 

TWO  LEFT,  V'ORST  IS  NOT  TOO  UAD 
FEATURl  NU.’IUI  R 1 WAS  USI  D 

FEATURE  MUnitt  R ? WA.S  USUI 

FEATURE  NtJItHFR  3 WAS  UM  0 

3 or  Tlir  INITIAL  3 WIRE  USED 

X & Y TOL:  .015077  .01!»:.80 

SUCCESS  ...  IlOCAll  WAS  ARI  I TO  RLACtI  lUF  TOI  FRANCES 
3 f I AlURI  S W1  RE  lltlEI) 

3 IIATCHES  WERE  USED 
THE  CHANCE: 

DX  & 2*sd(DX):  -.030304  .0150/7 

DY  & 2*sd(UY  : .080450  .010580 

Da  & 2*sd(l)a):  -3.71/137  1.018084 


Another  picture?  (Y  or  N):Y 
Picture:Wl xrC2.PlC 

Do  you  wnnt  to  see  each  feature's  match  as  it  is  found?  (Y  or  N):Y 
for  deliuoning  ...  set  hypIN]  to  V,  (fl  & V): 

Hov/  many  features  should  the  location  routine  start  with:  3 
<cr>  to  continue: 

VALUE  .815 
<cr>  to  continue: 

VAlllE  .7/8 
<cr>  to  continue: 

VALUE  .829 
<cr>  to  continue: 

THF  WORST: 

TWO  I EFT.  WOR.Sl  IS 
FEATURE  NIIMIIER 
FEATURE  HUFillLR 
FEATURE  NUMUEk 


2 

.0333/1 

. . 

TENTATIVE 

NOT 

TOO 

P.AfJ 

1 

WAS 

USE 

D 

2 

WAS 

USI 

D 

3 

WAS 

USI 

IT 

Pa^.c  2::’ 


V 


3 or  nil  itiniAi  3 wiui.  usld 

X & Y 101  ; .0‘.iS06/  .04/660 

NtW,R,A3-.K  ...  1 OK  X Alin  Y 

?.0)r.7?')  .os'^f)?/  .ii.r,."..'!) 

.661P.A4  .O'.OoW  .1.1041 

Tilt  TOltKAIiCL  COX  ON  lilt  SCKLlll  ( iti  pixels): 

116  l.'V  4.T 

r iqurc.  t’..o  .shoi's  the  (II  i (I i lull  tolc.rnnce  region  and  the  nrii 
• t <)  I I'lnnci  rcgiiiii  t nr  nin.r.itiir  d . lioticc  the  reduction  in  the 

si/r  of  the  aria  that  has  to  he  searched. 

VAllIC  .0?1 

<cr>  to  continue: 

I I A’ OKI  A IIA.S  CUM  AMill)  Pi  l•',^:/.:ll  111  L Y 

X A Y 101  : .0?0.’40  .04.T.T01 

success  ...  IlOCAir  was  ACir  to  KLACH  tup  TOLtKAIlCtS 

4 ILAIUKIS  WtPvL  IIIHC 
4 MAIC'HI.S  WIKI.  USI  1) 

TIIF  CIIANCI  : 

nX  A P*.Ml(l)X):  -.(.04048  .0P074O 

nY  ft  ?*.s(l(liY):  .P64547  .043.501 

Da  ft  2*sd(Uo);  -13.067337  1.506C4C 


Another  picture’  (Y  or  N):Y 
Picturo:wt  xr.c?.:>ic 

Do  you  v/.int  to  see  Ciicli  feature's  rruitcli  f.s  it  is  found?  (Y  or  N);Y 
***  for  (Irlunomo  ...  .set  hv)i[  Nl  t.f.  V,  (N  ft  V):33  +.006 

***  for  drlniO'lin.!  ...  set  liypINJ  to  V,  (II  ft  \/):36  +.006 

***  for  (IcIiiifiM in(i  ...  set  liypLlO  to  V,  (N  ft  V); 

I hr  sane  picture  is  used  for  this  execution  as  in  the  last 

cxanple,  hut  the  desired  precision  has  been  increased  from  .Ob 
to  .OOL). 

How  nnny  features  should  the  location  routine  start  v/ith;  3 
<cr>  to  continue: 

VAI Uf  .816 
<cr>  to  contiiuic: 

VAI  III.  .7/1'. 

<cr>  to  continue: 

VAI UP  .830 

<cr>  to  cont  inuc : lilt  WORST:  2 .033371  ...  TENTATIVE 

TWO  LfIT.  VOKSl  IS  NOl  100  K/.fi 

FLATIIKI  tUllllJI  K 1 WAS  LISI  U 

ttATUlU  NUIll’.l  K 2 V.'AS  LISI  h 

FE ATURf  mirillll!  3 WAS  USI  D 

3 01  TIIF  nil  UAL  3 WIRE  USLO 

X ft  Y TOl : .036067  .04/660 

HF  W,R,ft.3*rN  ...  f OK  X AND  Y 

2.016/24  .0,366;'/  .106881 

.6(,  188.3  .06064/  .16)641 

THE  TOl  FRANCE  BOX  ON  THE  SCREEN  (in  pixels): 

116  127  43  64 

VAI UE  .021 
<cr>  to  continue: 

FEATURE  4 HAS  BEEN  Aniil  D PLKMANFNTLY 

X ft  Y TOl  : .020240  .043301 

NEW,R.ft3*R  . . . FOR  X AND  Y 

.6263.31  .032608  .067624 

-.082668  .0.31660  .066876 

THE  TOIFRANCF  BOX  ON  Till  SCRl  t N (in  pixels): 

Oh  76  3o  44 

VAIUL  .86/ 

<cr>  to  continue: 

FlATUr.l  6 HAS  Ell  N AnntD  PEKnANliniY 

X ft  Y 101  : .0  336/0  .0  62802 

N[W,R.A3*P.  ...  ron  X AND  Y 

-.711020  .041026  .123086 
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.023521  .0'1//22  .H31G5 

THE  lOlEkANCL  BOX  ON  IIK  SCRLIN  (in  pixels): 

15  2(1  100  122 

VAlllE  .7-10 
<cr>  to  continue: 

IT  IS  TOO  FAR  on  TO  EVI  N TRY  IN  THE  FIT 
FEATURE  6 DIU  NOl  FIT  IN 

1 hr.  proyron  round  a mnt.ch  for  operator  6,  but  a prescreening 
showed  that  if  it  ive.re  included  in  the  fit,  it  would  not 
iviprovc.  the  precision  because  it  was  too  far  away  from  its 
predicted  posit  ion. 

X & Y TOE:  .03-1  .053 

Nl  W,R,&3*R  ...  FOR  X ANO  Y 
-1.177  .042  .127 

.910  .052  .157 

THE  TOLERANCE  BOX  ON  THE  SCREEN  (in  pixels): 

-9  2 63  75 
VAIUE  .744 
<cr>  to  continue: 

IT  IS  TOO  FAR  OFF  TO  EVI  N TRY  IN  THE  FIT 
FEATURE  7 DID  NOT  FIT  IN 
X & Y TOl  : .034  .0,53 

IlOCATE  COUID  NOT  MEET  HIE  Dl  SIRED  TOl  FRANCES 
***UNABLE  TO  DECIDE  Will  RE  IMI  OBJECT  IS*** 

Another  picture?  (Y  or  N):N 


End  of  SAIL  execution 
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VI.  CURVC5  A.S  TLATURCS 


C 111  vr^  <Tic  ini|irirt.iiii  viifliiii  |>ii^,i .(inm.il'li'  .’H^^cnibly  ticcsuse  several  sianciaid 
fall! icaiKPii  !i'cliiii']ucs  |.>iodiiCe  si.iootti,  ciirvui  siiriaces.  For  exaniijle,  drilling  a hole 
|)iodiiccs  a riicic.  which  a|i|)Oars  .as  an  cliijiic  in  .<  [nenne 

'I  his  a|i|iciidix  drsciihcs  a new’  tyjic  o:  0|ii-i a’c>i/fca:iiie  pair  based  upon  curves  The 
first  section  desciibcs  the  basic  laiionale  behinu  tin  u.'.e  of  curves,  the  second  section  describes 
the  type  o!  local  inidi niation  that  can  be  iisi'u  to  1 liter  out  incoria.ct  ruatebes;  the  third  section 
clesciibes  tei.hnn|ues  to  specify  cui  ves  v.  iilun  .a  \' V system,  and  the  tourth  section  discusses 
the  !>■  I >1  c sei itaiioii  cif  cm  vrs  ancl  evaluates  pai  aiu'  I'l  i/<  <1  pol'pioiuials  in  sonic  detail. 

An  cip<  I afoi /leafiii e pan  liased  iipCiii  linked,  nai  anisti  : I7' cl  polynomials  Iris  bei’ii 
impleineiited  and  has  been  used  wiiluii  the  \ V systei.i.  Given  an  initial  estimate  for  tbc 
|iCisition  cit  a cnive,  the  system  cm  antomaticall/  icfinc  ilic  model  ot  the  cuive  and 
characieii/e  its  usefulness  as  .m  opei .aior/U atm c p.air 


I 

t 


I 


Si  cl  ion  I 

i;,\Si(:  Ai’i’KOACii 

C)ni'  ot  il.e  b.oic  approaches  within  VV  is  to  locate  largo  features  that  ate  easy  to  find 
ill  Oiclei  to  help  loi  ate  small  feanins  that  ate  hauler  to  find  For  c\am|.ile,  when  visually 
SCI  voine  a seuw  info  .a  hole,  a V\'  pio  .iam  may  initially  Icicatc  a |iC)int  on  the  shaft  of  the 
scicwdiivci  .aiiel  ii'e  that  iiiloimatioii  to  help  Icieate  the  tijj  ot  the  sciesv.  C.onsidcr  tijuiie 
\'l  I I Fifiiii  N'l  I I .1  slniws  the  Sell  VV  oil  the  1 11. 1 oi  I hr  SCI  ovdi  1 Vi'i  as  it  is  a j)(j|  o.ic  1 1 ii  ig  1 1 n 
hole  I h<  c>\eil»v  iiidic  ill  s till  pi  mil' el  |ioei,iii|i  lee  the  lip  Of  the  sciew  and  the  initial 
tolriancr  n ion  .abniii  that  point  1 1 me  \ I I 1 b shovs  a March  pattrin  and  a point  on  tlic 
shaft  found  hy  an  nf  e O|'ia;oi.  1 igiiie  VI  1 1 c slio.  s the  nevv  lokiance  region  ahoiit  the 
tip  of  the  sen  w implieil  hy  the  infoi maiion  g;, lined  tioiri  the  match  ol  the  edge  operator. 
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♦ 
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I iguie  VI.l  1 b 


H'  M • ' ■ . 1 I’ll  ,, 

Figure  VI  1 l.c 


r 


C iiivcs  c.iii  I'l  ii'i'-l  III  i’:''''iiii  illy  ilii'  5.1. Ill  v.M^.  1 l.iviii  ■,  III'-. lied  .i  |Hiiiit  oil  ft  cm  VO,  emo 
c.m  niliico  till  lolii. 111(0  ir-  icm  .tlioii:  s I'l, ''iiic  o.  ii.u.'ir;:.  loi  cx:ini|ilc,  ccmsidci  tif'.iiio 
VI  1C,  uliiili  slni('s  I III  .'ll  I ,i5'i  I .il'ly  I'll  I ; (III  .1  : il'I'  1 lie  (.t'.il  o.'  tlic  t.i'k  is  to  loc.iio  iho 

luilr  Ill'll  till'  IS  III  III'  lei'll  l|(■ll:  li.iiiti  ci.'iiici.  i i^.mo  \'l  ICft  .sliciv.’s  iho  imii.tl  loloranco 

10:  loii  .il'i'Ui  lii.ii  lii'iO  (ilscii  .'I  |■ll^^i;o  ill'  ill'll, .c  .T.'.i'i'.ilji’y  .11  sor.ic  uiikiiowh  loCftiioii,  tlio 
0(1'*'.’  (i|"i.'tii'i  cm  111'  n'lilii.il  III  II  I'.iiiii  [iiii'iiiii  III  iiy  III  loc.'iic  it  |ioiiit  oii  the  Iftir*,*.*  ciiivi,’. 
l i'  1110  \ I 1 '^*1'  'linu's  siiiii  .1  Si  .til.  1 1 |i.i:ii';ii  fiiii.  n i.i  .'.i.iiiri  , I'lOiiu  Hint  locntly  ri[)|ic.iis  in  be  h 
|iiiiiii  ('ll  till  111.  I.  cm  vi  li  iK.'it  is  ,1  |".iiiit  Oil  111  liti  cuisc.  ulicio  would  the  holt  hole  hi;' 
l i'.iiio  \ 1 I c siiiivs  the  cm  VO  |ii'si:ii''iii ni  sci  ili  ii  (1)  the  cm  so  I'.'issos  thiciugli  the  riiiitchiii;|'' 
[iciiiit  .'tiul  ('.!)  iho  sl'/jio  o',  the  cliivo  i.iii'.ih'ii  ttio  slejic  01  the  niiitchtiig  (joint.  It  the  cdHO 
o|iei:iti)i  I'loiliiii's  ex.iit  v.iiiies  Ion  tin'  (lu'iriiin  n;  tho  rii.i tc 1 1 in o,  jjoirit  snei  the  slo()e  of  the 
curve,  iiii'.l  if  theii'  is  no  ani'.ul.n  nnco; Minty  nssijci.itcd  with  the  location  of  the  binKe 
.assen'il '1', , tho  bolt  hiili'  winilol  be  .it  the  |io;.itio,i  slifiwn  in  ii^.mt  \ 1 1 Cc. 

} ri-  e c'j "'I  .Ill'll s air  not  |iii.rise  imii  tii.  .i  is  usually  snnie  unceitaiiuy  rifnjut  the 
Cl  I' lit ,tt i"i I (if  tin  (iiiit.  1 lii'so  uiii.oi t iiiitios  (.,iii  i”  ci.iiiibirioC'  to  toiiii  a new  tolei'ance  lOf  ioii 
.'ibeiiit  till'  Ill'll  1 III  I s .iinjile,  11  the  in  ii'ii:,i; i(„i  i.i  ll.i  hi.il.e  ,'i"i  mbly  is  Only  l.no'wn  within 
I'liis  cu  liiinns  lilt' in  (hioi's^  tl,'  bolt  Ih'I'.'  v ill  In  on  ilio  iic  of  the  curve  shov.n  in  Iniiro 
1 riel  ft  till  I'-iiiniito  (it  the  sli'jii  ('ll  I'. 'll, "0,.'  f'V  iJi.’id  e e(|fc,' .voi,  )(,((  an  imo  il.iintf  ot  livi 
cb'-  ioos,  til''  cm  VO  III  I','  III  liK.iio..!  .t!  .'my  |ie''iii"ii  i lun  ih  i.in  .o  shown  in  tivuro  VI  1 C e 

And  liii.'illy.  It  thfii  IS  an  uncoi i.nnty  o;  one  (U'.i!  al.ciut  the  i",  itiun  of  the  iriaichiti;';  (jciiiii. 

till'  nii|ilii'd  tolei.iiico  1 Os, loll  wciiild  hi  i\|i,inio\!  sli  htly.  1 lyuro  VI  l.C.f  shows  the  final 
tiller. tiico  ii  ion  almiit  the  hoi'  |iiOiiU('ii.  by  ill  (it  iho>i'’  nncoi tainties  .Notice  timt  the  new 
tub  1. (lice  ii  ion  IS  considei ahly  sm.'iller  th.m  the  cm  inal  fine.  This  (irocess,  which 
II ic I c 1 1 lo I it.i II V bmlds  tho  III  w tnl.  iancr  ii  . ion  tiniii  the  iJill'  iont  iincei'iainties,  woiks  the  best 
wbeii  the  micoi  l.nnlios  iii  o mainly  twu-dir.ien;.ion.il 

1 liis  ii(linit|iK  IS  only  cinr  of  ilnee  nO'Sibl'’  techniiiuos  to  incorporate  the  infoi ivi.iticin 
[oiineci  fiool  a |iiiin'  on  a cm  ve  (.i)  ciit'ci  iikhi  j'oi  .ii;i-'n  into  the  leasi-siiuai es  tiiiini’,  routiin''. 
(b)  tvio-dimoi).sion.'il  niodilliiiy,  of  tie  iincoit.iintii  .and  (c)  stati.stic.il  modelling  fium  seveial 
li  a II line  |ii('im  1 s 'I  bo  tin ■ apjii h n hes  covei  a i .m  i fi orn  the  analytic  to  the  ex|)ei  imental 

1 lie  |iosiiii'in  anil  oi  i'.nt  I'luii  infoi  nrition  .'iss.ici.ited  ivith  a point  on  a siiii|»le  cm  vc, 
such  as  a ciule,  c.in  he  liiii.cily  inciu |'o: an d inm  tie  le.ist-siiu.iics  fitting  routine.  Mote 
Cl  II  oij  ilio  <fi  1 1 , III  Vi  s ,n  • iniin.  dilliiiil.'  hi.o.(i/se  it  IS  hatciei  to  s(iccity  the  ei|n.itions  that  iilate 
the  Icio.ition  (il  the  imve  to  the  (m'.itiuii  and  nt e nt.iiiuii  intorm.iiion  (uodneed  hy  the  edi'  e 

li|  e I ,<ti  II  . 

If  thi  cmvc  is  cHi.ipliS  aii'l  tie  unci : tainties  chain, e the  I'wo-cliiiicnsional  a|i|je.'it ance 
of  the  curve,  ii  is  siili  (inssilile  to  o.ii.ict  a ceit.iiii  ai'iioiint  of  iiifoi mation  from  a (loint  on  the 
cmve  by  rniihlliiiS’.  lie  two  cime  nsicinal  disii  ibiiiinii  of  irnpliei.  (lositions  foi  the  point  of 
inieiest  'I  h.'it  is,  dmin,g  the  naming,  slagL',  loc.iii.'  .t  (lOiii:  on  the  cuivc,  determine  the  ini(jlicd 


I 


t 
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(josition  for  the  point  of  interc5t.  locate  the  actual  position  of  the  point  of  interest,  and  gather 
statistics  on  the  errors  between  the  implied  and  actual  positions.  At  execution  time  these 
statistics  can  be  used  to  produce  a tolerance  region  about  the  implied  position  for  the  point  of 
interest 


•Section  2 

LOCAL  TESTS  FOR  A POINT  ON  A CURVE 


♦ 


Given  an  analytic  curve  and  a match  of  an  edge  operator,  is  the  matching  edge  a point 
on  the  curve?  Put  another  way,  what  tests  can  be  applied  to  the  local  information  in  order  to 
filter  out  incorrect  matches’’  Essentially  the  same  types  of  tests  can  be  applied  to  decide  if  an 
edge  is  on  a curve  as  to  decide  if  an  edge  is  on  a line.  The  tests  are  based  upon  five  types  of 
information: 


(1)  contrast, 

(2)  distinctness, 

(?)  slope, 

(4)  continuity, 
and  (5)  curvatiiie. 

When  an  edge  operator  is  used  to  locate  a point  on  a line,  its  parameters  are  set  to  locate 
potential  edges  that  have  a ceitain  contrast  and  distinctness.  Elaving  found  a candidate  that 
meets  these  qualifications,  the  candidate  is  checked  to  make  sure  that  it  is  within  the  correct 
range  of  angles  For  cxam|)lp,  if  the  line  is  vertical,  the  candidate  edge  is  horizontal,  and  the 
angular  unceitainty  is  only  twenty  degrees,  the  candidate  edge  can  not  be  on  the  line.  If  the 
candidate  edge  is  within  the  correct  range  of  angles,  its  local  continuity  can  be  checked.  To 
perform  this  test  the  edge  operator  is  applied  on  either  side  of  the  candidate  edge.  If  there 
arc  no  edges,  the  candidate  is  discarded.  If  the  new  edges  are  at  inconsistent  angles  or 
positions,  the  candidate  edge  is  rejected  because  its  local  curvature  is  incorrect 

The  generalization  from  lines  to  curves  is  straightforward.  The  edge  operator  is  tuned 
to  locate  candidate  edges  that  have  a certain  contrast  and  distinctness.  The  range  of 
acceptable  angles  includes  the  complete  set  of  slopes  along  the  curve  and  is  augmented  by  the 
angulai  uncertainty  associated  with  the  part.  For  example,  if  the  slope  of  the  curve  varies 
from  130  degrees  to  270  degrees,  and  if  there  is  a ten-degree  uncertainty  associated  with  the 
object  upon  which  the  curve  is  based,  the  total  range  of  the  acceptable  angles  is  120  degrees 
to  280  degrees  The  continuity  test  is  the  same  for  curves  as  it  is  for  lines.  The  curvature  test 


checks  the  ;iii  les  |iusiiioiis  of  the  iv.,i  or  thii.  c.lc’.o  I'Oiiits  used  in  the  conlitmity  t< '•t.  H 
they  arc  no:  consisn-nt  with  tlie  cnivanm;  oi  the  analytic  curve,  the  candidate  edge  is  rejected. 
For  c\rinii<lc.  considet  the  hi.ikc  .I'n-mi'ly  t\nnrj.l.’  discussed  in  the  jjicvious  section  it  the 
cd;‘“e  cn'eiaiMi  Icic airs  a c.andidati.  i'd<  e v.  itli  the  C'lieei  conti ast,  di>iinctness,  and  an^le,  and  it 
the'  edee  oj'eiate)i  is  .ii'idicri  once  on  cither  MCtC  (»1  the  candidate  tu  check,  the  continuity  and 
cinvatino.  \\h.»t  do  the  icsnlts  shown  in  ti^  ure  VI  C l iriijdy?  1 hey  air|jcar  to  iinidy  that  the 
curv.itiire  at  the  oiij.inal  candidate  edge  is  f.reatrr  tlnn  the  curvature  of  the  laigc  curve. 
'I  hiis.  the  edge  is  iJiuh.ihly  on  the  sni.dl  cuive  loirncd  by  the  bolt  hole,  not  the  large  curve. 


•• 
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I'lguie  Vi. C l 

^ his  ty|je  of  checking  is  rcirriecl  tci  as  /I'Ci?/  cliccking  because  the  decisions  ate  based 
upon  a small  poiiion  of  the  picuire.  A curve  may  be  two  hundred  pixels  long,  but  the 
decision  whether  or  not  the  candidate  edge  is  on  tlie  curve  is  based  upon  the  intensities  in  a 
small  aiea.  such  as  a 10x10  array  of  pixels.  Olol'al  consistency  checking  is  performed  by  the 
least-squares  culliiag  routine. 


Seel  inn  3 

SI-lClIl  iCATlON  0!  A ClIRVF 


III  the  cuiifiit  implinientatiiMi  of  tl.e  V\'  sysicrri  a |iro;', rammer  can  specify  a cuive 
featuie  hy  pointing  cnit  a few  points  near  the  curve  in  a planning  picture  and  letting  the 
system  automat irally  locate  and  characi<  i :/>'  tiu  cu:  ve  that  actually  appears  in  the  pictuic. 
This  interactive  pioccss  involves  the  following  step': 


(1)  Use  a ciiisor  to  indicate  a few  points  ne.ir  the  curve  Possibly  include 


Pat;e 


Pi(;ure  VI. 3.1. f 
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the  appioximate  slo|)cs  at  the  indicated  points. 

(2)  1 lave  the  system  use  an  edge  o|>eia!or  to  search  for  the  closest  edge  tn 
the  picture  to  each  indicated  point. 

(3)  Fit  an  analytic  curve  through  the  actual  edge  points. 

(I)  Have  the  system  follow  the  curve  as  far  as  it  can  in  order  to  pick  up  as 
much  as  possible  of  the  actual  curve  in  the  picture. 

(5)  Have  the  system  walk  along  the  complete  curve  and  gather  statistics  on 
(a)  the  distance  between  each  actual  edge  point  and  the  closest  point  on 
the  analytic  curve  and  (b)  the  distance  between  each  actual  edge  point 
and  the  point  on  the  analytic  curve  that  has  the  same  slope. 

(6)  If  the  statistics  indicate  that  the  analytic  curve  does  not  closely 
approximate  the  actual  curve,  refine  the  analytic  curve  by  incorporating 
more  of  the  actual  edge  points. 

1 he  automatic  refinement  and  characterization  of  a curve  mentioned  in  steps  5 and  6 
make  it  possible  for  the  user  to  specify  the  desired  range  of  residual  errors  in  adVance.  The 
system  stops  refining  the  curve  as  soon  as  the  predicted  residual  errors  are  sufficiently  small. 
The  residual  errors  are  important  because  they  determine  the  size  of  the  tolerance  region 
implied  about  another  feature,  given  one  point  on  the  curve. 

Consider  figure  VI  3.1,  which  shows  a partially  completed  chainsaw  engine.  The  goal 
is  to  locate  the  dark  screw  hole  th.at  is  marked  in  figure  V]  3. l.a.  The  user  indicates  the  two 
|)oints  and  associated  slopes  shown  in  figure  V1.3.1.b.  The  system  locates  the  actual  edge 
points  in  the  picture  and  fits  an  analytic  curve  through  the  points  (as  shown  in  VI.3.1  c).  It 
then  follows  the  curve  until  it  loses  the  curve  in  the  shadow  at  the  lower  right  end  and  until 
it  finds  a dtamatic  change  in  cuivatiire  at  the  upper  left  corner  (sec  figure  V1.3.1.d).  Then 
the  system  walks  along  the  complete  curve  gathering  statistics.  It  decides  that  the  middle  of 
the  curve  needs  to  be  better  appro.ximatecl  by  the  analytic  curve,  so  it  adds  another  point  to 
the  analytic  curve  fitting  routine.  The  refined  curve  is  shown  in  figure  V1.3. 1.e.  Given  the 
new  curve,  the  potential  tolerance  region  about  the  screw  hole  implied  by  one  point  on  the 
curve  IS  shown  in  figure  V1.3.i.f. 

The  only  reason  that  this  technique  is  referred  to  as  an  interactive  method  instead  of  an 
automatic  method  is  that  the  user  has  to  point  out  a few  points  near  the  curve.  If  that 
subtask  could  be  done  automatically,  the  whole  suggestion  process  could  be  done 
automatically.  How  can  the  system  automatically  suggest  potentially  useful  curves?  If  the 
system  had  a three-dimensional  model  of  the  objects  in  the  scene,  it  could  produce  a synthetic 
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picture  (possibly  just  a lux:  cli  rr.s'iiig,),  wliicli  i:  could  .malyzc  for  long,  distinct  curves.  For 
e.xanyplc,  consider  ligiiic  VI.J.2,  wlucli  sliows  a huldcn-lint  drawing  of  the  expected  scene 
when  the  chainsaw  engine  is  at  its  planning  location. 


1 he  ctiiient  system  is  a first  step  lov.ard  suoh  an  automatic  featuie  suggestion  system. 
1 he  iisei  inrci actively  defines  the  three  dimensional  models  oi  the  objects  in  the  scene.  '1  he 
system  aniomaiically  produces  tlie  liidden-line  chawing  in  which  each  curve  is  appro.viinatcd 
by  a seiiuence  ol  line  segments  'I  he  system  rn.airitains  an  internal  description  that 
distinguishes  between  the  line  segments  that  are  used  for  this  type  of  ap|)roximation  and 
those  that  rcpicscnt  actual  line  segments,  'I  he  user  iritei actively  points  out  potential  curves  in 
the  line  drawing.  The  system  auiornaiically  locales  and  characterizes  the  curve  that  actually 
a|)pears  in  the  planning;  picture. 
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Section  4 

KEPRESENTATION  OF  A CURVE 


I 


1 he  fiiM  three  sections  of  this  apiK.nciix  have  described  the  use  of  curves  assuming 
that  a representation  for  curves  exists  that  makes  it  possible  to  compute  the  necessary 
information,  such  as  the  length  of  the  curve.  Tfiis  section  briefly  describes  desirable 
properties  of  such  a representation  and  discusses  possible  representations. 

Desirable  properties  of  the  rc|)resentation  include- 

(1)  The  representation  should  be  able  to  model  a wide  variety  of  curves, 
such  as  ellipses  and  French  curves. 

(2)  There  should  be  a way  to  locate  the  point  or  points  on  a curve  that  have 
a certain  slope 

(3)  It  should  be  easy  to  move  a curve  around,  e.g.,  translate  it  and  rotate  it 
in  the  plane  of  the  picture. 

{4)  There  should  be  a way  to  slide  along;  the  curve  a certain  distance. 

(5)  It  should  be  easy  to  change  the  form  of  a curve  incrementally. 

(6)  There  should  be  a way  to  dcteimine  the  length  of  a curve. 

(7)  There  should  be  a way  to  locate  the  closest  point  of  approach  to  a curve 

from  a given  point 

There  aie  several  possible  ways  to  represent  curves  Some  satisfy  these  requirements 
better  than  others.  Four  reasonable  possibilities  arc; 

(1)  linear  segments. 

(2)  parameterized  polynomials, 

(3)  regular  splines, 
and  (I)  li-spliiics 

Linear  segments  are  simple,  but  a large  number  of  points  are  needed  to  approximate  a curve 
Parameterized  polynomials  require  fewer  points  than  linear  segments,  but  they  are  very 
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sensitive  to  ttic  slope  of  tlie  cm  vc  wlun  the  curve  is  tilruost  linerif.  Regular  splines  provide  a 
nice  range  of  curves,  but  they  arc  diiiicult  to  mociify  locally,  li-splines  can  be  locally 
inoclified,  but  they  lequiie  a gre.it  deal  of  coiiipuuiiion  to  locate  |ioiri:s  with  matching  slopes 
or  the  point  of  closest  apjuoach.  1 hus.  all  of  the  alicrnatives  have  their  advantages  and 
disadvantages. 

1 he  representation  tfiat  h.is  been  ii-,i|ilemcnie'l  is  a piecewise  polynomial  The 
lepieseniation  is  an  oidercd  list  of  poin!-slu|.cs  i hat  is,  each  entry  in  the  list  has  a 
two-diniension.il  position  and  a .slo()e.  'I  hc  cu.  ve  is  the  simplest  curve  that  passes  through 
the  [joints  and  has  the  correct  slojie  at  the  two  points  (As  such,  it  is  a simple,  parameterized 
spline  through  the  points.)  1 he  rt|.iiesent.ition  of  the  .arc  between  two  point-slojies  is 


(VI. 1)  X(t)  = A*t  + BU  + C (0.0  $ I i 1.0) 

? 

Y(  t ) = li*t  + [ Jit  + I . 

which  fits  a parabola  between  two  jiomt'slijjies 

The  coefficients,  A through  F.  can  be  computed  fiom  two  point-slope  pairs.  (x0,y0),s0 
and  (xl,yl),sl  Let  S(t)  icpieseni  the  slojic  of  the  curve  at  the  position  (X(t),Y(t)). 
Then  S(t)  can  be  computed  as  follows 

(I  Y(t) 


dt 

(VI. 2)  S(t)  = 

d X(t) 


Cit 

or 

2*D*t  a F 

(VI. 3)  S(l)  = 

2*A*t  a B 


Then  the  six  Ciiiiations  that  relate  the  cociiicicnts  to  the  point-sloijC  values  are: 
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(VI. 4) 


xO  = X(O.O) 
yO  = Y(O.O) 
SO  = S{0.0) 
xl  = X(l.O) 
yl  = Y(l.O) 
si  = S(l.O) 


(VI. 5) 


xO  = C 

yO  = F 

sO  = £/B 

xl  = A + B + C 

yl  = 0 ♦ E ♦ F 

si  = (2*0+E)/(2*A't-B). 


Tlie.se  six  equations  can  be  solved  for  A through  F to  produce  the  following  set  of 
computations  for  the  coefficients; 


(VI. 6) 


2*sl*(x2  - xl)  - 2*(y2  -yl) 

B - 

(s2  - si) 

A - (x2  - xl)  - B 
C - xl 
E ► sl*B 

D ► (y2  - yl)  - E 
F *•  yl. 


It  is  also  a straightforward  computation  to  determine  the  point  on  a curve  segment  that 
has  a specific  slope.  Let  s be  the  slope  in  question.  Then  s and  the  curve  parameter,  t are 
related  by  the  following  equation: 


(VI. 7) 


Solving  (VI. 7)  for  t produces 


2»D*t  ♦ E 


2*A*t  ♦ B 


(VI. 8) 


s*B  - E 


{2*D  - 2*A*s) 


1 tins,  tin’  pom;  with  a slope  oi  s is  ( X(  t ) . Y(  t ) ).  w hrn;  t is  ^.iven  by  (VI  .8 ) 

'Mil'  ii  piesi'iit.ition  iei]mirs  only  .1  h v.’  poiiii-slopes  to  icpirseiit  a coinples.  ciii  ve 
pn'cof'Iy.  le,  sti  tliat  tin;  position  ana  cniv.ituri,  01  the  aiialytic  etiive  closely  matches  the 
po'.iiioii  aiiil  cinvaime  ol  the  atmal  cm  ve  1 01  e\  <c,i|)le,  consiciei  fit.ure  Vl-i  I.  It  shows  a 
pictme  01  a cm  ve  and  the  analytic  ajipiu'.im  inon  to  the  cm  vc  loimeci  fitini  seven 
pomt-sliiprs  lie.  si\  aus'i  'I  he  ciiive  can  h'  easily  improved  by  including  an  additional 
point  Litiw'ei.n  twci  oi  thecmieiit  [loinis. 

I I he  m.im  di.iwhacks  to  this  upi e.sem.moii  me  that  (I)  thcie  arc  some  degenerate  cases 
that  aiise  when  the  slopes  arc  veihral,  {?.)  infie.::ion  pfiints  have  to  be  interactively  pointed 
out  (a  jMi.ihol.i  Can  not  tit  throii;;h  an  infirction  point),  and  (.'’•/ the  computations  requirea  to 
line!  the  pciint  ol  cli'oest  approach,  etc  mciei'-e  lineail)  as  the  number  of  point-slopes  used  to 
appiOMinatc  the  curve  incieases. 


